Implementing Unique IDs in GraphQL

12 min readMar 31, 2020

The GraphQL docs are uncomfortably non-specific about the requirements for the id field. They say that the id should be unique, but they don’t really say how unique or why it matters. Should id values be universally unique, or is it good enough that they be unique among objects of the same type? To make matters more confusing, GraphQL client implementations implement a well-motivated trick that silently “fixes” some of the obvious incorrect choices.

If you don’t feel like reading, the quick answers are:

The id field is supposed to be unique across all objects handled by the client.
In practice, it is usually sufficient if the id field is unique-per-type, because GraphQL client caches concatenate the __typename field with the id field by default. In order for this to work, you need to make sure that __typename is sent across the wire.

Globally unique id values are the better practice for reasons I’ll describe below.

Why is a Unique ID Needed?

Let’s start with why the id field needs to be unique. The id mainly gets used in two scenarios:

To fetch a specific object from a server (if the protocol supports this)
To uniquely identify objects for caching on the client. This lets objects be stored in the client cache just once.

The Fetch Case

In many cases, the fetch-from-server case can be handled with an id value that is unique per type. This is true because a query or a mutate usually is working on specific types of objects, and you can use this context to know what type of thing to fetch or store on the server.

The exceptions are cases where you have what amounts to a union type; an argument value or a return value that can be more than one object type. One example is an API argument that might could be either a user or a group. From a design perspective, this is a dubious practice.

While you can get away with an id that is unique-per-type for this most of the time, it’s not “future proof.” At some point in the future, you’ll run into one of three problems:

You’ll need a union type for backwards compatibility, often to work around a design failure in your protocol.
You’ll need to ship around an object that came from someplace else. Such objects may nothave a natural __typename field.
You will need to implement a second wire protocol using something that isn’t GraphQL. Most of the wire protocols don’t assume that objects have a __typename field, so they assume that id values are globally unique.

Save yourself some hassle later. Implement a globally unique id value.

The Client Cache

The client-side cache in GraphQL doesn’t actually know anything about your objects. If an id field is present, it uses this to uniquely identify the object it has received. That way, if two queries return the same object, it will only get stored once in the client-side cache. Objects can get sent back and forth several times as your application receives query responses. For example, the user may navigate back to an existing view, with the result that a query is performed a second time. The application-side memory footprint goes down a lot if duplicate objects can be unified. This also helps the client-side UI give a consistent picture of object updates.

Contrary to most of the documentation, GraphQL caches are two-level caches. The first level associated a query and its parameters with a set of unique object ids. The second level stores the objects. If two queries return the same object, the cache will only store it once. Note that what happens if the two queries return different fields on the object isn’t specified! That’s a frequent cause for caching-related bugs in GraphQL clients. Since GraphQL caching is the subject of another note, I’ll leave off on that here.

How Client-Side IDs are Generated

On a GraphQL client, a unique ID is computed for every object to identify it in the client cache. Most GraphQL clients use three strategies to come up with a unique ID for the objects they store:

Use a programmer-provided function. This is set in the dataIdFromObject field when you create a cache. At that point you’re in complete control.
Concatenate the __typename and the id field to form a client-side unique lookup key for object caching. This is why the addTypeName attribute of a client side cache defaults to true. It forces all request to include the __typename field in every fetched object (if it isn’t already present) so that it is available to form the unique lookup key.
If the object doesn’t have an id field, the path from the query to the object is used instead. This is really bad; it might well be better not to cache these objects at all.

So how should you generate id field values on the server side? The answer, of course, is “it depends.”

Server Options for Creating ID Values

There are several options for generating ID values sensibly. It’s even possible to use a mix of these if you wish.

Option 1: Rely on __typename + id For Uniqueness

This is the simplest approach, as it’s more or less built in to GraphQL, and it plays nicely with auto-incrementing IDs in a SQL database that are only unique within a table.

The “catch” is that the actual id field in the object won’t be unique. If the id value gets copied around, or if you have an id field somewhere that can refer to objects of different types (that is: a union id field), that can lead to trouble. If your database supports sequence objects, the simplest way to resolve that is to generate your ids using a sequence object so that they are unique across tables. The catch with that is that an ID value that gets passed back to the server doesn’t encode any idea about which table should be searched to find the correct object.

Another way to solve this is to add field resolvers for each of your types on the server:

// Useful to be able to read the IDs for debugging:
const clearIDs: Boolean = true;
// ...
function EncodeID(typeName: String, id: String) {
  if (clearIDs)
    return `${typeName}-${id}`;
  else
    return Buffer.from(`${typeName}-${id}`).toString('base64');
}function DecodeID(encodedID: string | null | undefined) {
  // Inbound ID can be null in cases where a field is not defined.
  // Allow that case to pass without incident:  if (encodedID == null)
    return null;  if (isUndefined(encodedID))
    throw new UserInputError(`DecodeID(): id argument is undefined, got ${encodedID}`);  let decoded = Buffer.from(encodedID, 'base64').toString('ascii');  if (clearIDs)
    decoded = encodedID;
  return Number(decoded.split('-')[1]);
}//...
const resolvers : Resolvers = {
  //...
  ID: GraphQLID,
  //...
  User: {
    id: function(parent, args, context, info) {
      return EncodeID('Build', parent.id);
    },
  },
  //...
};

This trick works in both JavaScript and Typescript. You’ll need to remove the type annotations for JavaScript. Note that if the client concatenates the __typename value, the key on the client will get bigger, but it will still be matched properly.

This provides a useful debugging feature, but it can expand the wire traffic significantly. The place where it is actually needed is if you have an id field value that can refer to objects of more than one type. If you do that, you’ll need to be storing something that tells you where to look for the object. At that point, you either need to concatenate something that differentiates the type, or you need to build a side table that tells you where to look.

Option 2: Use Sequence Objects

If your database supports sequence objects, and you are using integer unique ids, use a sequence object to generate them! You may not need the uniqueness now, but it costs essentially nothing, and if you turn out to need uniqueness later you’ll be glad you have it.

There are two problems with sequence numbers:

They are guessable, which means that you can’t rely on the uniqueness of the id values for any sort of access control decisions.
When you someday need to merge data sets, you’ll discover that the two data sets have overlapping id values.

The latter is actually a reason to generate universally unique id values rather than just values that are unique to your application.

Option 3: Use a GUID/UUID

This is a widely advocated option, and I believe that it’s the right way to go. Old-style GUIDs are basically big random numbers, which have a small chance of collision. Today, most programs have switched to UUIDs, which rely on various uniquely assigned hardware identifiers to actually guarantee uniqueness. UUIDs tend to be issued in sequential blocks. They are unique, but they are also guessable.

GUID/UUID values are large when they are represented in text form. A text UUID is 36 bytes. But watch out for UTF encoding schemes! If your table is set up using utfmb4 as the collating sequence, and you store the GUID string in your database as varchar(36), those characters will be expanded to 4 bytes each. Pretty soon your index won’t fit in memory any more, which will drastically reduce query performance. And as with all user-provided unique IDs, the database will do a lookup before inserting to check that the ID value is actually unique. This will limit insert performance.

If you are considering using a GUID/UUID, convert them to 16 byte binary values as recommended by this application note. Converting UUIDs from a 36 byte string to a 16 byte integer for storage will simultaneously help them fit in the cache and speed up comparisons in the index. While the note describes a method that is specific to MySQL/MariaDB (but see uuid_to_bin() and bin_to_uuid() in MySQL), the same idea can be used in SQL Server and Oracle.

Where to Generate Unique IDs

Unique id values should not be generated on the client. The mechanisms available to the client to ensure uniqueness aren’t strong enough. The right place to generate unique id values is at the server. This has consequences for both protocol design and application logic.

Unique id values should not be generated by your database. Doing so makes it hard to store object graphs (even simple ones), and it adds the cost of a round trip to the database for every unique id you generate. Some databases generate them well, others do not. Better to have control of this and actually know that you can rely on your unique values — you’re going to have a lot invested in their uniqueness by the time this is done.

Protocol Concerns From a protocol perspective, you need to distinguish between create operations (which need to assign a unique ID) and update operations (which don’t). You can either create separate create and update operations in your protocol, or you can declare that an update operation that passes a “zero UUID” is actually a create operation. Either one is reasonable. Most databases still don’t implement “create-on-update” support, so you can’t necessarily merge these operations in your back end code.

Application Concerns From an application perspective, UUIDs generated on the server means that round trips may be needed to store complex things. Sending a complex graph of de novo objects is actually pretty rare. In my experience, the more common case happens when you create an object that contains an “owned” list of something. I’ve chosen to handle this by writing create operations that pass the containing object and the list as separate parameters. Since the list is owned, we know on the server that both the list and the container need to be created.

This leaves the graph case. If your graphs are acyclic, one option is to accept the round trips to the server to get the UUIDs generated. They probably aren’t as bad as you imagine, and this approach makes it pretty simple to handle “live update” scenarios across multiple clients.

The other two options are:

Have the server send you a batch of “fresh” UUID values and insert them on the client side. You’ll need to distinguish between create and update operations again, and the server will have to confirm that the values are actually unique before it stores them.
Assign client-unique values, send the graph over, and have the server replace the client-unique values with server-unique values before storing the items. This is clean, but it can be complex to implement.

Keep in mind that any design where the server may be forced to hold a large object set in memory is a potential source of denial-of-resource attacks. Round trips may be the lesser evil.

Further Observations About Binary GUIDs

There are a bunch of notes out there (including the MariaDB app note linked above) that suggest you should binary-encode the UUID for storage (for the reasons above) and then decode it back to hexadecimal for transmission and comparison. Unless you need a conventional UUID string for external reasons, this is silly — the binary version is exactly as unique (or as random) as the string version. You need to convert it to a string to keep GraphQL happy (because the id field is defined as a string type), but that can be done using base 16 encoding (hexadecimal, so 32 characters), base 64 encoding (22 characters), base 10 encoding (40 characters) or base 85 encoding (hard and not well-motivated). The id values are going to get passed around a lot, so encoding them to reduce their size is well-motivated.

In our applications, we needed a UUID generator anyway. We’ve written a small piece of server-side code that does this transparently at UUID allocation time.

Update: Handling Database Views

When I originally wrote this note in March of 2020, everything I had done fit nicely into an object-relational mapping approach. Sooner or later you run into things where a view becomes appropriate. At that point you have some decisions to make about row identity. The answer depends on the design pattern.

Filters

Sometimes views get used as filters. Perhaps you’ve built a materialized view or a prepared query that only incorporates “live” orders. If all you are doing is building a filter, you aren’t changing the identity of the underlying objects. You should use the existing unique identity of the object.

Object Assembly

Sometimes it may be convenient to assemble a single object from multiple (correlated) tables. Perhaps you want to add locally managed metadata to an externally sourced object. In this case, the same object id should be used in all tables, and it should be used as the object id returned from the view.

Summaries

This is the really powerful use case of views (or really: queries). You want, say, a summary of results by quarter or week. In this situation, you really don’t want to be shipping all of the data to the client. You want to build the summary on the server. A view captures a frequently used query, but more importantly, it serves to define the type of the answer.

A summary row isn’t an object; it’s a purely computed result that will never be updated from the client side. In abstract, a row number makes a fine object ID in a view, though there are two cases to watch out for:

If the sort order can at the query can be modified from the client (watch out for pagination here), you need to do something more clever. As long as the lines are unique and unchanging, hashing the objects may work nicely. This continues to work if new objects are added, but it doesn’t work if existing objects are modified (it breaks the client caching strategy).
If the content of the rows can change, you may need to consider what fields constitute the primary key for each row and build a hash on those. This requires more discipline, but it’s actually a better approach than hashing the entire object. This remains friendly if the GraphQL client ends up requesting fields selectively — it’s still going to have to request the id field in any case.

UUIDs for the Cloud

If you are building a cloud-based service, you are likely to end up with multiple components/containers that need to assign new UUIDs. This is a lot more complicated than pulling UUIDs out of a library implementation, because you really need the results to be unique across all of the containers. All of the scalable solutions involve allocating UUIDs in a separate set of (replicated) containers and having the consumers allocate them in blocks.

Two observations:

Replicated allocation preserves UUID uniqueness, but there is no solution that preserves monotonicity. This is true because successive allocations may be served by different allocators. If your application relies on monotonicity, you’ll need to remove that reliance.
It will be tempting to solve this by switching to randomly allocated identifiers (GUIDs). Resist! Not because GUIDs are bad in and of themselves, but because (a) random number generation is hard to do well, and (b) when the randomness is good, the rate of random number generation isn’t very high. The number of npm GUID generation packages that naively rely on a bad random number source is impressive.

The catch in this is that you’re going to end up with multiple containers allocating UUIDs for you, and you want to make sure that they allocate non-colliding UUIDs. The most common implementation is for the generator to rely on the machine’s MAC address as a source of uniqueness. This doesn’t work so well in virtual machine environments or container environments. There are a couple of reasonable options to look at:

Use a random number generator to generate a unique 48 bit seed instead of using the MAC address. If you know the number of replicates you can do this statically.
Expose the pod UUID to the container and use the most rapidly changing bits of the pod UUID instead of the MAC address.
If you are talking about human-generated objects, you could consider using Twitter’s Snowflake IDs instead of UUIDs. This works mainly because humans don’t type very fast.

I don’t care for the Snowflake approach because the IDs are not globally unique. Someday you are going to have to merge two databases, and you’re going to discover that re-mapping unique IDs during merge is (a) a royal pain, and (b) has a way of breaking cross-references. Best to make sure they are unique in the first place. An ounce of prevention…

A detailed design is beyond the scope of this note. I’ll be building a UUID generator service for Kubernetes shortly, and I’ll add a link to it here when it is public.