Learning About Large Json Files

When connecting to different APIs you might come across some large datasets that might not fit in normal text editors.  This makes it difficult to learn about how the data is structured.

There exists a really cool tool called jq that makes it a little easier to see the data structure.  It takes a while to learn the syntax, but it is pretty powerful.  Below are a few of the commands I use regularly to learn how my data is structured.

Assume we have a file names largeResponse.json.  That has a similar structure like:

{
  "data_available": true,
  "query": "The query is here",
  "results": {
    "details": [
      {
        "id": 1,
        "data1": "data1",
        "data2": "data2"
      },
      {
        "id": 2,
        "data1": "data1",
        "data2": "data2"
      }
      ... Many more rows here
    ]
  }
}

Getting the keys

 jq 'keys' largeResponse.json

This command will list all of the keys of the root object.  We can also see the keys of child objects using

jq '.results | keys' largeResponse 

Assuming there is a child object of 'results'  this will show the list of the keys in the results object.

Get an object

jq '.results' largeResponse.json 

will show all of the results object, which for this case would be a a LOT of information.  However, for smaller objects, like the query, it could be very helpful.

Working with Arrays

With a large list of data it is useful is see a sample of the data.  

Length. We can see the length an array with something like this

jq '.results.details | length' largeResponse.json

Notice how details is an array.

Get a single Object.  We can get an item from an array with the index.

jq '.results.details[0]' largeResponse.json

This would return 

{
  "id": 1,
  "data1": "data1",
  "data2": "data2"
}

Get a range of objects.  jq can also get a range in an array.

jq '.results.details[0,10]' largeResponse.json

This would return the first ten objects in the details array.

Tons more

And of course you can a million other things that I haven't come close to exploring.  To see all that you can do take a look at the jq manual.

Misc

To print out multiple fields:

jq '.users[] | .first + " " + .last'

Creating a JSON Schema from a POJO

If you're documenting your REST API it might be useful to show the schema definition behind the JSON objects being used.  The JSON Schema format provides a definition similar to XML's XSD.

It isn't very difficult to generate a JSON schema with Jackson.  Given a simple POJO with some properties which includes a list of another simpler POJO:

public class SimplePojo {

 private Integer id;
 private String name;
 private String description;
 private List children;
 // getters and setters ommitted
}

public class SimplePojoChild {

 private Integer id;
 @JsonProperty("cName")
 private String childName;
 @JsonProperty("cDescription")
 private String childDescription;
 // getters and setters ommitted
}

We can use the Jackson JsonSchema Module to generate the Schema with code similar to this:

public class JsonSchemaGenerator {

 public static void main(String[] args) throws JsonProcessingException {
  ObjectMapper mapper = new ObjectMapper();
        SchemaFactoryWrapper visitor = new SchemaFactoryWrapper();
        mapper.acceptJsonFormatVisitor(SimplePojo.class, visitor);
        JsonSchema schema = visitor.finalSchema();
        System.out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(schema));

 }

}

This code would generate this json schema:

{
  "type" : "object",
  "id" : "urn:jsonschema:com:bmchild:model:SimplePojo",
  "properties" : {
    "children" : {
      "type" : "array",
      "items" : {
        "type" : "object",
        "id" : "urn:jsonschema:com:bmchild:model:SimplePojoChild",
        "properties" : {
          "cName" : {
            "type" : "string"
          },
          "id" : {
            "type" : "integer"
          },
          "cDescription" : {
            "type" : "string"
          }
        }
      }
    },
    "name" : {
      "type" : "string"
    },
    "description" : {
      "type" : "string"
    },
    "id" : {
      "type" : "integer"
    }
  }
}

The only downside is if you're using a different version of jackson (we were using one from the codehaus repo) you'll need to change your json annotations so it will print out correctly.

All the code can be found on My Github Account

Spring MVC 3: Property referenced in indexed property path is neither an array nor a List nor a Map

JQuery's $.ajax does an excellent job mapping a json object to parameters, but when you start getting into more complex objects Spring MVC doesn't know how to interpret it sometimes.

For example if you have a json object like:

{
   "beans":[
      {
         "agreemendId":1,
         "answerId":2
      }
   ]
}

JQuery will map your parameters like

beans[0][agreementId]:1
beans[0][answerId]:2

The problem is that Spring MVC is expecting a parameter format like

beans[0].agreementId:1
beans[0].answerId:2

In order to get that you can do it in 1 of two ways. You can do the quick and dirty way, which changes the way you're building your json object. Or, the other way is to extend the jQuery plugin to build parameters differently.

To change the javascript code was pretty simple and looked something like this

var answers = {};
answers['beans[' + index +'].agreementId'] = agreementId;
answers['beans[' + index +'].answerId'] = value;

To modify the jquery plugin I would suggest taking a look here.

And for reference here are the pojos I was mapping to.

SpringMVC 3 Serialize enums to JSON

Apparently Jackson and SpringMVC 3 doesn't know how to handle enum properties when serializing to json, however there is a solution and this stack overflow thread has it.

Adding Jackson to a SpringMVC project for JSON mapping

Here are the dependencies you'll need for SpringMVC to automatically map objects to JSON objects.



And here is an example of a method in a controller (Note the returning type is a simple pojo and your request should specify "application/json" as the accept type):


UPDATE - And it also looks like this also works as an alternative to the above dependencies: