CHAPTER 4 Encoding and Evolution

Everything changes and nothing stands still.

Applications inevitably change over time. However, in a large application, code changes often cannot happen instantaneously:

1. With server-side applications you may want to perform a rolling upgrade (also known as a staged rollout), deploying the new version to a few nodes at a time, checking whether the new version is running smoothly, and gradually working your way through all the nodes.
2. With client-side applications you’re at the mercy of the user, who may not install the update for some time.

This means that old and new versions of the code, and old and new data formats, may potentially all coexist in the system at the same time. In order for the system to continue running smoothly, we need to maintain compatibility in both directions:

1. Backward compatibility: Newer code can read data that was written by older code.
2. Forward compatibility: Older code can read data that was written by newer code.

Formats for Encoding Data

Programs usually work with data in (at least) two different representations:

1. In memory, data is kept in objects, structs, lists, arrays, hash tables, trees, and so on. These data structures are optimized for efficient access and manipulation by the CPU (typically using pointers).
2. When you want to write data to a file or send it over the network, you have to encode it as some kind of self-contained sequence of bytes (for example, a JSON document).

Thus, we need some kind of translation between the two representations. The trans‐ lation from the in-memory representation to a byte sequence is called encoding (also known as serialization or marshalling), and the reverse is called decoding (parsing, deserialization, unmarshalling).

Many programming languages come with built-in support for encoding in-memory objects into byte sequences. For example, Java has java.io.Serializable, Ruby has Marshal, Python has pickle, and so on.

Moving to standardized encodings that can be written and read by many program‐ ming languages, JSON and XML are the obvious contenders.

Binary Encoding

For data that is used only internally within your organization, there is less pressure to use a lowest-common-denominator encoding format. For example, you could choose a format that is more compact or faster to parse.

Apache Thrift and Protocol Buffers (protobuf) are binary encoding libraries that are based on the same principle. Both Thrift and Protocol Buffers require a schema for any data that is encoded.

message Person {
    required string user_name       = 1;
    optional int64  favorite_number = 2;
    repeated string interests       = 3;
}

Thrift and Protocol Buffers each come with a code generation tool that takes a schema definition like the ones shown here, and produces classes that implement the schema in various programming languages

With type schema specified, we only need to encode values which is more compact.

So, we can see that although textual data formats such as JSON, XML, and CSV are widespread, binary encodings based on schemas are also a viable option. They have a number of nice properties:

1. They can be much more compact than the various “binary JSON” variants, since they can omit field names from the encoded data.
2. The schema is a valuable form of documentation, and because the schema is required for decoding, you can be sure that it is up to date (whereas manually maintained documentation may easily diverge from reality).
3. Keeping a database of schemas allows you to check forward and backward compatibility of schema changes, before anything is deployed.
4. For users of statically typed programming languages, the ability to generate code from the schema is useful, since it enables type checking at compile time.

Modes of Dataflow

Dataflow Through Databases

Databases, where the process writing to the database encodes the data and the process reading from the database decodes it

Dataflow Through Services: REST and RPC

RPC and REST APIs, where the client encodes a request, the server decodes the request and encodes a response, and the client finally decodes the response

Message-Passing Dataflow

Asynchronous message passing (using message brokers or actors), where nodes communicate by sending each other messages that are encoded by the sender and decoded by the recipient