Cacheable JSON Results and Schemas

by Erich Ocean (onitunes)

This is a collection of notes I wrote a while back for my own backend server implementation, which supports both SproutCore, Cocoa, and native iPhone clients, and generates the model files and schema stuff automatically. I’m posting it here in the hopes that it well help give others some useful ideas for their own projects.

The JSON relational object format

or How to send partial object graphs from the server

Okay, when sending back a JSON object, I had intended on doing nothing special for relationships. I now think that’s a mistake. Suppose we have an object, Apple, with a to-one relationship to object Orange. Here’s the JSON response for an Apple object:

<pre>{ "id": "1234567890abcdef1234567890abcdef", "name": "sample attribute", "$orange": { "id": "1234567890abcdef1234567890abcdef" } } </pre>

So, rather than do:

we do the full object reference:

Thus, an object reference is any JSON object with an “id” attribute that is either (a) 32 hexchars long, or (b) is at least three characters long, contains no spaces, and has a colon (‘:’) character not at the beginning and not at the end.

NOTE: In my JSON results, properties prefixed with ‘$’ refer to other objects, e.g. they’re like foreign keys in SQL.

The primary advantage I see to using objects as references is that it allows us to eager-include object attributes (and other object references), e.g.

<pre>{ "id": "1234567890abcdef1234567890abcdef", "name": "sample attribute", "$orange": { "id": "1234567890abcdef1234567890abcdef", "name": "another simple attribute", "$grower": { "id": "1234567890abcdef1234567890abcdef" } } } </pre>

I think this will make it easier to recognize objects, since they’re always JSON “objects” with an “id” property. Here’s the corresponding result when the relationship is null:

<pre>{ "id": "1234567890abcdef1234567890abcdef", "name": "sample attribute", "$orange": null } </pre>

And here are the four versions for to-many relationships:

<pre>{ "id": "1234567890abcdef1234567890abcdef", "name": "sample attribute", "$oranges": 0 }

{
“id”: “1234567890abcdef1234567890abcdef”,
“name”: “sample attribute”,
“$oranges”: [
{ “id”: “1234567890abcdef1234567890abcdef” },
{ “id”: “1234567890abcdef1234567890abcdef” }
]
}


So, the “destination” for a to-many relationship is never null; it can, however, be 0 (zero) or an empty array, and you can either include the id array or not. If you do include the id array, you can also include the full object (as usual), recursively.

It’s possible to determine the kind of relationship simply by types: null or object is to-one, number or array is to-many. This allows for a limited form of “self-discovery”: the schema is self-defined by the objects in it.

Structs can be included by embedding them as objects with a key name that does not begin with a dollar symbol. Structs should also include a uuid key to identify themselves. The schema for the class is required to know what the struct represents, since unlike objects, structs are not self-identifying.

The JSON relational object schema

The JSON object format only includes persistent attributes and relationships. To get access to computed attributes and fetched relationships, you need to ask for the entity’s schema. This can be done directly by interpereting the id attribute.

There are two forms of the id attribute. The first is the 32 hexchar key. The first 8 hexchars reference the type, and the remaining 24 hexchars are the primary key. The second form is the “entity:primary-key” pair. The characters before the colon are the type, the characters after the colon are the primary key.

In either case, you can make a request to “/schema/” to get back a description of the remaining (shared) relationships for the entity. Here’s a sample response for /schema/entity/12345678:

<pre>{ "schema": "1234567890abcdef1234567890abcdef", // the schema this entity is a part of "id": "12345678000000000000000000000000", // the actual class "object" (do a GET to get its attributes) "entity": "Task", "parent": null, "abstract": false, "protocols": [], "class-name": "AOTask", "class-properties": { "attributes": { "name": {} }, "computedAttibutes": { "capitalizedName": [ uses: "name", returns: "String", server: true, clients: [ "SproutCore", "Cocoa" ] } } }, "properties": { "attributes": { "name": {} }, "relationships": { "orange": { "kind": "to-one", "dependent": false, "inverse": "task" } }, "computedAttibutes": { "capitalizedName": [ uses: "name", returns: "String", server: true, clients: [ "SproutCore", "Cocoa" ] } }, "fetchedRelationships": {} }, "fetchRequests": [ "index" ] // i.e. queries you can make against the entity itself } </pre>

This has a trivial computed type, capitalizedName, which does what it says, and changes when the name property changes.

There are two ways to make use of computed attributes and fetched relationships. The first is server-side. Your client code needs only to know when to request the attribute (which is why the “properties” key is there). It can then simply do a request to get the new “computed” version of the property, e.g. /1234567890abcdef1234567890abcdef.capitalizedName would return the (computed) capitalized name. The primary advantage of server-side handling is that no code is required on the client. The downside is the round-trips to the server, which obviously increases latency.

Fetched relationships are (by definition) always server side. So, it’s only computed attributes that could benefit from the (optional) speed up.

The other way to do this is client-side; however, loadable code for this purpose must be available. You can load the code for an entire schema by doing /schema.<type>, e.g. /schema.js would give you the schema’s code in JavaScript format. You can also load individual schemas by doing /schema/init.<type> followed by /schema/12345678.<type>. It is expected that /schema/init.<type> loads “boostrap” code that allows individual entities to be correctly loaded one-by-one.

Given that there are a lot of entities, it makes sense to load them all at once during production.

I suppose we could support dates with the following convention:

This works because we don’t currently allow the String type as a valid type of attributes/relationships that begin with a dollar sign. A null (unknown) date would need to be “unknown”. We can’t use null, because we already use that for empty relationships.

Using Varnish Cache and <esi:include/> for caching

The whole idea behind Varnish caching is to have Varnish cache only stable URLs in JSON object format, either single-object or an array of JSON objects (depending on the specific URL).

In both cases, only JSON data is stored, with <esi:include/> directives in the “array” case.

The “object” case looks like this:

<esi:include src="/1234567890abcdef1234567890abcdef/1234567890abcdef1234567890abcdef"/>

The “array” case looks like this (/object/transaction):

<pre> [ <esi:include src="/1234567890abcdef1234567890abcdef/1234567890abcdef1234567890abcdef"/>, <esi:include src="/1234567890abcdef1234567890abcdef/1234567890abcdef1234567890abcdef"/>, <esi:include src="/1234567890abcdef1234567890abcdef/1234567890abcdef1234567890abcdef"/>, <esi:include src="/1234567890abcdef1234567890abcdef/1234567890abcdef1234567890abcdef"/>, <esi:include src="/1234567890abcdef1234567890abcdef/1234567890abcdef1234567890abcdef"/>, <esi:include src="/1234567890abcdef1234567890abcdef/1234567890abcdef1234567890abcdef"/>, <esi:include src="/1234567890abcdef1234567890abcdef/1234567890abcdef1234567890abcdef"/>, <esi:include src="/1234567890abcdef1234567890abcdef/1234567890abcdef1234567890abcdef"/>, <esi:include src="/1234567890abcdef1234567890abcdef/1234567890abcdef1234567890abcdef"/>, <esi:include src="/1234567890abcdef1234567890abcdef/1234567890abcdef1234567890abcdef"/> ] </pre>

Basically, Varnish will substitute the (probably cached) JSON objects, one for each <esi:include/> in the array. The result is a valid JSON array literal that can be eval()’d on the client side.

The beauty of this approach is that objects can always be cached because they are only ever cached with their permanent URL, which will never change. This is true even when the object isn’t cacheable from the point of view of the browser.

NOTE: For relationship queries, the transaction of the left hand side is used in the permanent URL. The database always either returns a single JSON object, or an <esi:include/> reference to a single JSON object, or an <esi:include/> reference to an array of JSON objects.