Learning About Large Json Files

When connecting to different APIs you might come across some large datasets that might not fit in normal text editors.  This makes it difficult to learn about how the data is structured.

There exists a really cool tool called jq that makes it a little easier to see the data structure.  It takes a while to learn the syntax, but it is pretty powerful.  Below are a few of the commands I use regularly to learn how my data is structured.

Assume we have a file names largeResponse.json.  That has a similar structure like:

{
  "data_available": true,
  "query": "The query is here",
  "results": {
    "details": [
      {
        "id": 1,
        "data1": "data1",
        "data2": "data2"
      },
      {
        "id": 2,
        "data1": "data1",
        "data2": "data2"
      }
      ... Many more rows here
    ]
  }
}

Getting the keys

 jq 'keys' largeResponse.json

This command will list all of the keys of the root object.  We can also see the keys of child objects using

jq '.results | keys' largeResponse 

Assuming there is a child object of 'results'  this will show the list of the keys in the results object.

Get an object

jq '.results' largeResponse.json 

will show all of the results object, which for this case would be a a LOT of information.  However, for smaller objects, like the query, it could be very helpful.

Working with Arrays

With a large list of data it is useful is see a sample of the data.  

Length. We can see the length an array with something like this

jq '.results.details | length' largeResponse.json

Notice how details is an array.

Get a single Object.  We can get an item from an array with the index.

jq '.results.details[0]' largeResponse.json

This would return 

{
  "id": 1,
  "data1": "data1",
  "data2": "data2"
}

Get a range of objects.  jq can also get a range in an array.

jq '.results.details[0,10]' largeResponse.json

This would return the first ten objects in the details array.

Tons more

And of course you can a million other things that I haven't come close to exploring.  To see all that you can do take a look at the jq manual.

Misc

To print out multiple fields:

jq '.users[] | .first + " " + .last'