This is the Month to Learn jq

1/4/2018
·
#learn-one-thing
#jq

If JavaScript is the language of the web, then JSON is its dominant pidgin—self-documenting, simple, and dolled up with all the curly-bracing we’ve come to expect from C-style syntax. Sure, purpose-built serialization protocols like Thrift and Cap’n proto are faster and more efficient, but for public APIs, JSON remains the state of the art.

There are plenty of tools available to poke at JSON documents from the comfort of your browser or a familiar IDE. But while GUI tools are convenient for initial exploration, screenshots are a terrible medium for document or reproducing changes we might make.

That’s where jq (https://stedolan.github.io/jq) comes in: for analysts, operators, developers, and anyone else that routinely works with JSON data, it’s the best way to not edit JSON data by hand. Used effectively, it will simplify data analysis and save you time.

To whet your appetite, let’s point jq at the Star Wars API:

$ curl -sL swapi.dev/api/vehicles?search=AT- \
  | jq keys
[
  "count",
  "next",
  "previous",
  "results"
]
$ curl -sL swapi.dev/api/vehicles?search=AT- \
  | jq '.results[0] | keys'
[
  "name",
  ...
  "url"
]

Now that we know a bit about the API, let’s transform the data into an Excel-ready CSV. One small adjustment to the filter and we’re there.

$ curl -sL swapi.dev/api/vehicles?search=AT- \
  | jq -r '.results[] | [.name, .url] | @csv'
"AT-AT","https://swapi.dev/api/vehicles/18/"
"AT-ST","https://swapi.dev/api/vehicles/19/"
"AT-TE","https://swapi.dev/api/vehicles/53/"
"AT-RT","https://swapi.dev/api/vehicles/76/"

One line gives us the result and a clear explanation of how it was produced. And that’s jq for you. Write short, reusable filters and just add JSON. Voila!

Filter syntax may feel a bit opaque at first, but there’s an interactive tutorial to introduce the basics. From there, move on to the official manual and the user-contributed cookbook. In no time at all, you’ll have a powerful new tool for exploring and transforming JSON data.

Why wait? There’s no time like now to up the data-analysis game with jq!

Featured