This blog explores the story behind adding type information behind the
Neo4j Query API. “What’s the Query API?” I hear you
ask. Its a easy and simple way to query Neo4j through your favourite HTTP client. The Query API is
now available on Aura (in beta).
Discord in the Driver’s channel.
Neo4j’s First Interface
HTTP support for Neo4j was one of the first interfaces exposed for accessing Neo4j data
(after the Embedded API). However,
the HTTP interface has somewhat lagged behind the preferred transport bolt
which powers
the official and community drivers.
Over the last few years that HTTP support has been slowly rejuvenated, with new functionality being added to the
existing Cypher Transactional HTTP API such as
limited support for clustering and HTTP/2 support.
The latest improvement is a bigger change.
The API as well as its type format has been completely redesigned. For now, it available as “Neo4j Query API”
in parallel to the existing API. The Query API supports two different formats:
application/json - the API’s default, which maps Neo4j results directly onto JSON, choosing the most suitable JSON type.
This format allows you to quickly query your data with minimal tooling.application/vnd.neo4j.query, which defines a new mapping of Neo4j types onto JSON aimed at more
complex interactions with results.
This post focuses on the typed result format of the new API and assumes a basic understanding of the API.
If you’d like to get a grasp of the basics before reading this
post, check out the wonderful post by the project’s PM Jonathan Giffard.
The Problem
All good solutions must start by correctly identifying the problem. In JSON,
the de facto standard for data exchange via HTTP, we have the following native types:
null- boolean (
true or false)
- string (
"A mighty string")
- number (
1, 12.34)
- array (
["a", 123, "bunch", true, "of", ["huh"], "stuff"])
- object (
{"im": "an object"})
In Neo4j we have a much richer type system.
Neo4j has types that JSON also has including: Booleans, Strings and Maps, but it also has things that JSON does not,
such as: Nodes, Dates and Points.
Note: this is a simplified view of the how the types intersect and there are some difference
we omit here for brevity.
The problem we have is that in order to interact with Neo4j via HTTP’s preferred type format JSON, we have
to introduce a mapping between the two type systems. Since the underlying types between the two systems diverge, and JSON is the
format that is missing the additional types, we have to extend JSON so that Neo4j types can be communicated without
losing the benefits of the additional Neo4j types.
Why deal with Types at all?
Types also come with operations on these types. For example, if you know you are
dealing with a Node type, then you also know that is has labels that can be retrieved from it (i.e an
operation of labesl() -> [String]. Each of Neo4j’s types come with many useful operations which makes further
processing the results possible.
But didn’t you say the Query API support plain JSON?
Yes, but to do this we had to transform the richer Neo4j types directly onto the JSON types, losing their richness, and
making writing a parser much more difficult. In the application/json format we serialize types such as points into the
most suitable string representation
(in this case Well-known text representation of geometry).
This format is best suited for simple use cases such as: ad-hoc querying, rendering directly to a UI or where it’s not possible
to use additional tools or dependencies to further process the response. On the request side, we also would like
to insert these Neo4j types into Neo4j and,
while this is possible with cypher functions,
it would be much better to explicitly use these types in our requests.
Now that we have fully identified the problem, we can address what a good solution to it looks like.
We set ourselves the following design goals for the new format:
-
Human Readability - One of the great strengths of JSON is that it is understandable to read without
any extra tooling (save a pretty formatter). This is great for ad-hoc querying since no further processing is needed.
It also makes developing with it much easier since you can fire off a request and examine it before solidifying it in code.
-
Machine Readability - As discussed, we want to preserve the richness of Neo4j types and this means in practice
processing the results and deserializing them into the native (or custom) types of the calling language.
We will expand on this in a later post but for now we understand this goal to be summarized as “Can I easily write a
parser with standard tooling in mainstream programming languages?”.
-
Streamability - There’s no reason to restrict the size of data that the Query API can return.
Since it might take a while to complete the returning of a large result, it would be great if the results were returned
in a way that they can be processed by the client as they become available (i.e. to not wait for the
whole thing comes back before processing it).
-
Flexibility - Even after over a decade and a half of development, Neo4j is still rapidly changing with new features such as
CDC and
RAG support. Most important for this API is that
this format needs to be able to adapt to new types.
Performance is a consideration, but, as you can figure from these criteria, the main goal is ease of use.
Neo4j’s performance offering is covered by our in-house binary protocol Bolt
which power the official drivers.
Adding Types To JSON
A great starting point on adding type information to JSON can be found in Peter Hilton’s
Blog
but most options here
essentially add a type properties at various points the JSON. The question is where should we put it? We decide to use
Option 3 of having type paired with value in an JSON object representing the results:
{
"type":"Integer",
"value":"1"
}
Let’s look at how this solution stacks up against our requirements:
-
Human Readability - Whilst, as Peter points out, some reduction in readability from the additional nesting of values (this may not look
too bad in this example but remember this would multiply for each nested value), it is still readable
and intuitively it is clear which what is a type and what is a value. Mostly-Check.
-
Machine Readability - a parser would only need to check the value of what is in the type field
and so this lead to a simple if block. But there’s no need to completely write a parsers from scratch. Many parsing
libraries in many different languages support this style of adding type information making writing a parser quite easy.
Check.
-
Streamability - Since types and values are nestled nicely together there’s nothing that prevents us from streaming
values independently of each other. Check.
-
Flexibility - Adding a new type is as simple as introducing a new String for that type. Check.
We introduce a type/value pairing object which can be returned when adding
application/vnd.neo4j.query as the Accept header:
curl --location 'http://<your_server>/db/neo4j/query/v2' \
--header 'Content-Type: application/vnd.neo4j.query' \
--header 'Accept: application/vnd.neo4j.query' \
--header 'Authorization: Basic <your_auth_token>' \
--data '{"statement": "RETURN 1"}'
{
"data": {
"fields":["1"],
"values":[{
"$type":"Integer",
"_value":"1"}
]},
"bookmarks":["..."]
}
Wait!? What’s with the $ and _? In general it’s good to make it clear when a JSON property is part of the result and
when a property is considered meta-data and these two prefixes make it clear these properties are to be treated differently.
We could exclude these but Neo4j is also in the unfortunate position that we would have a clash of names with Neo4j’s node
type having a property called type. Best to avoid the confusion and potential mistakes.
And that’s all that’s to it. Of course we now have to define what each of the types’s string tag and the format of
the value but we leave this for our documentation.
The problem we describe earlier in this blog also applies when we want to push data into Neo4j. The lack of types
available to us in JSON make it impossible for Neo4j to understand what type you are trying to store. For example take
the following request:
curl --location 'http://<your_server>/db/neo4j/query/v2' \
--header 'Content-Type: application/json' \
--header 'Accept: application/vnd.neo4j.query' \
--header 'Authorization: Basic <your_token_here>' \
--data '{
"statement": "RETURN $zonedDateTime",
"parameters": {"zonedDateTime": "2015-11-21T21:40:32.142Z[Antarctica/Troll]"}
}'
{
"data":
{
"fields":["$zonedDateTime"],
"values":[
{
"$type":"String",
"_value":"2015-11-21T21:40:32.142Z[Antarctica/Troll]"
}]
},
"bookmarks":["..."]
}
It’s impossible for Neo4j to know that you wanted a ZonedDateTime and not a String. It is possible to tag the Cypher statement
with dateTime() procedure and then it will be read as the correct type but then you miss out on the security and performance benefits
of parameterising your queries. Fortunately you can still parameterise your query by using application/vnd.neo4j.query as
input by setting Content-Type header with that value:
curl --location 'http://localhost:7475/db/neo4j/query/v2' \
--header 'Content-Type: application/vnd.neo4j.query' \
--header 'Accept: application/vnd.neo4j.query' \
--header 'Authorization: Basic <your_token_here>' \
--data '{
"statement": "RETURN $zonedDateTime",
"parameters": {"zonedDateTime": {"$type": "ZonedDateTime", "_value":"2015-11-21T21:40:32.142Z[Antarctica/Troll]"}}
}'
{"data":{"fields":["$zonedDateTime"],"values":[{"$type":"ZonedDateTime","_value":"2015-11-21T21:40:32.142Z[Antarctica/Troll]"}]},"bookmarks":["..."]}%
{
"data":
{
"fields":["$zonedDateTime"],
"values":[
{
"$type":"ZonedDateTime",
"_value":"2015-11-21T21:40:32.142Z[Antarctica/Troll]"
}]
},
"bookmarks":["..."]
}
Give application/vnd.neo4j.query a try today on Aura and be sure to reach out with questions
or feedback on Discord.
Acknowledgements
In this first post I thought it would be fitting to mention many folks who have helped to make the Query API a reality.
My fellow colleagues Michael Simons and Gerrit Meier for the initial PoC that kicked
off this whole thing and their continued support throughout the project. Dmitriy Tverdiakov who helped out with a key
piece of architecture. Grant Lodge who has reviewed many of my nonsensical PRs. Antonio Barcelos who helped to validate
this type format by writing the first non-java parser in javascript. A big thanks to Waiariki Koia and the folks in the
Aura who are helping to bring this show onto the Aura stage. Stefano Ottolenghi for the great work putting the
docs together.
Andy Heap and Ivan Fulöp for their patience and support during
some sticky moments. As mentioned already the project’s PM driving this whole thing
forwards Jonathan Giffard. And last but not
least Richard Macaskill who was the previous wonderful and kind PM on this project before he sadly passed away last year.
]]>
