This is a prerelease version.

View latest

Compact Serialization

As an enhancement to existing serialization methods, Hazelcast offers Compact Serialization, with the following main features.

  • Separates the schema from the data and stores it per type, not per object which results in less memory and bandwidth usage compared to other formats

  • Does not require a class to implement an interface or change the source code of the class in any way

  • Supports schema evolution which permits adding or removing fields, or changing the types of fields

  • Can work with no configuration or any kind of factory/serializer registration for Java classes and Java records

  • Platform and language independent

  • Supports partial deserialization of fields, without deserializing the whole objects during queries or indexing

Hazelcast achieves these features by having a well-known schema of objects and replicating them across the cluster which enables members and clients to fetch schemas they don’t have in their local registries. Each serialized object carries just a schema identifier and relies on the schema distribution service or configuration to match identifiers with the actual schema. Once the schemas are fetched, they are cached locally on the members and clients so that the next operations that use the schema do not incur extra costs.

Schemas help Hazelcast to identify the locations of the fields on the serialized binary data. With this information, Hazelcast can deserialize individual fields of the data, without reading the whole binary. This results in a better query and indexing performance.

Schemas can evolve freely by adding or removing fields. Even, the types of the fields can be changed. Multiple versions of the schema may live in the same cluster and both the old and new readers may read the compatible parts of the data. This feature is especially useful in rolling upgrade scenarios.

The Compact serialization does not require any changes in the user classes as it doesn’t need a class to implement a particular interface. Serializers might be implemented and registered separately from the classes.

It also supports zero-configuration use cases by automatically extracting schemas out of the classes and Java records using reflection, which is cached and reused later, with no extra cost.

The underlying format of the Compact serialized objects is platform and language independent.

Using Compact Serialization With Zero-Configuration

Compact serialization can be used without a registering the serializer in the member or client configuration. Hazelcast tries to extract a schema out of the class, using reflection. If successful, it registers the reflective serializer associated with the extracted schema and uses it while serializing and deserializing instances of that class. If the automatic schema extraction fails, Hazelcast throws an exception.

Currently, Hazelcast supports extracting schemas out of classes that have the following field types:

  • Primitive types: boolean, byte, char, short, int, long, float, and double

  • Wrapper classes of primitive types: Boolean, Byte, Character, Short, Integer, Long, Float, and Double

  • String

  • java.time.LocalDate, java.time.LocalTime, java.time.LocalDateTime, and java.time.OffsetDateTime

  • java.math.BigDecimal

  • Enums

  • Array, List, ArrayList, Set, HashSet, Map, HashMap of the types shown above

  • Nested classes that contain the fields above and arrays of them

Assume that you have the following Employee class.

public class Employee {
    private long id;
    private String name;

    public Employee(long id, String name) {
        this.id = id;
        this.name = name;
    }

    public long getId() {
        return id;
    }

    public String getName() {
        return name;
    }
}

If you don’t perform any kind of configuration changes and use the instances of the class directly, no exceptions are thrown. Hazelcast will generate a schema out of the Employee class the first time you try to serialize an object, cache it, and reuse it for the subsequent serializations and deserializations.

The same holds true for the Java records. Hazelcast supports serializing and deserializing Java records, without an extra configuration as well.

Assuming the Employee class above were a Java record:

public record Employee(long id, String name) {
}

The following code would work for both of them.

HazelcastInstance client = HazelcastClient.newHazelcastClient();
IMap<Long, Employee> map = client.getMap("employees");
Employee employee = new Employee(1, "John Doe");
map.set(1L, employee);
Employee employeeFromMap = map.get(1L);

Implementing CompactSerializer

Another way to use compact serialization is to implement the CompactSerializer interface for a class and register it in the configuration.

For example, assume that you have the same Employee class. Then, a Compact serializer can be implemented as such.

public class EmployeeSerializer implements CompactSerializer<Employee> {
    @Override
    public Employee read(CompactReader reader) {
        long id = reader.readInt64("id");
        String name = reader.readString("name");
        return new Employee(id, name);
    }

    @Override
    public void write(CompactWriter writer, Employee employee) {
        writer.writeInt64("id", employee.getId());
        writer.writeString("name", employee.getName());
    }

    @Override
    public Class<Employee> getCompactClass() {
        return Employee.class;
    }

    @Override
    public String getTypeName() {
        return "employee";
    }
}

The last step is to register the serializer in the member or client configuration.

Programmatic Configuration:

SerializationConfig serializationConfig = new SerializationConfig();
serializationConfig.getCompactSerializationConfig()
        .addSerializer(new EmployeeSerializer());

Declarative Configuration:

  • XML

  • YAML

<hazelcast>
    ...
    <serialization>
        <compact-serialization>
            <serializers>
                <serializer>
                    com.example.EmployeeSerializer
                </serializer>
            </serializers>
        </compact-serialization>
    </serialization>
    ...
</hazelcast>
hazelcast:
  serialization:
    compact-serialization:
      serializers:
         - serializer: com.example.EmployeeSerializer

A schema will be created from the serializer, and a unique schema identifier will be assigned to it automatically.

From now on, Hazelcast will serialize instances of the Employee class using the EmployeeSerializer.

Schema Evolution

Compact serialization permits schemas and classes to evolve by adding or removing fields, or by changing the types of fields. More than one version of a class may live in the same cluster and different clients or members might use different versions of the class.

Hazelcast handles the versioning internally. So, you don’t have to change anything in the classes or serializers apart from the added, removed, or changed fields.

Hazelcast achieves this by identifying each version of the class by a unique fingerprint. Any change in a class results in a different fingerprint. Hazelcast uses a 64-bit Rabin Fingerprint to assign identifiers to schemas, which has an extremely low collision rate.

Different versions of the schema with different identifiers are replicated in the cluster and can be fetched by clients or members internally. That allows old readers to read fields of the classes they know when they try to read data serialized by a new writer. Similarly, new readers might read fields of the classes available in the data, when they try to read data serialized by an old writer.

Assume that the two versions of the following Employee class lives in the cluster.

class Employee {
    long id;
    String name;
}
class Employee {
    private long id;
    private String name;
    private int age; // Newly added field
}

Then, when faced with binary data serialized by the new writer, old readers will be able to read the following fields.

public Employee read(CompactReader reader) {
    long id = reader.readInt64("id");
    String name = reader.readString("name");
    // The new "age" field is there, but the old reader does not
    // know anything about it. Hence, it will simply ignore that field.
    return new Employee(id, name);
}

Then, when faced with binary data serialized by the old writer, new readers will be able to read the following fields. Also, Hazelcast provides convenient APIs to check the existence of fields in the data when there is no such field.

public Employee read(CompactReader reader) {
    long id = reader.readInt64("id");
    String name = reader.readString("name");
    // Read the "age" if it exists, or use the default value 0.
    // reader.readInt32("age") would throw if the "age" field
    // does not exist in data.
    int age;
    if (reader.getFieldKind("age") == FieldKind.INT32) {
        age = reader.readInt32("age");
    } else {
        age = 0;
    }
    return new Employee(id, name, age);
}

Note that, when an old reader reads data written by an old writer, or a new reader reads a data written by a new writer, they will be able to read all fields.

CompactSerializationConfig

The configuration can be used to register either

  • an explicit CompactSerializer

  • a reflective serializer for a class to override other serialization mechanisms.

In case of an explicit serializer, you have to supply a type name for the class in the serializer.

Choosing a type name will associate that name with the schema and will make the polyglot use cases, where there are multiple clients from different languages, easier.

When a class is serialized using the reflective serializer, Hazelcast will choose the fully qualified class name as the type name automatically.

Below is the way to register an explicit serializer for a certain class.

  • XML

  • YAML

  • Java

<hazelcast>
    ...
    <serialization>
        <compact-serialization>
            <serializers>
                <serializer>
                    com.example.FooSerializer
                </serializer>
            </serializers>
        </compact-serialization>
    </serialization>
    ...
</hazelcast>
hazelcast:
  serialization:
    compact-serialization:
      serializers:
        - serializer: com.example.FooSerializer
SerializationConfig serializationConfig = new SerializationConfig();
serializationConfig.getCompactSerializationConfig()
        .addSerializer(new FooSerializer());

Lastly, the following is a sample configuration that registers reflective serializer for a certain class, without implementing an explicit serializer.

This way, one can override other serialization mechanisms for a certain class such as Java serializable.

  • XML

  • YAML

  • Java

<hazelcast>
    ...
    <serialization>
        <compact-serialization>
            <classes>
                <class>com.example.Bar</class>
            </classes>
        </compact-serialization>
    </serialization>
    ...
</hazelcast>
hazelcast:
  serialization:
    compact-serialization:
      classes:
        - class: com.example.Bar
SerializationConfig serializationConfig = new SerializationConfig();
serializationConfig.getCompactSerializationConfig()
        .addClass(Bar.class); // Uses the fully qualified class name as the type name

If you want to override the serialization mechanism used for Serializable or Externalizable classes and use Compact serialization without writing any serializer, you must add those classes to the configuration.

GenericRecord Representation

Compact serialized objects can also be represented by a GenericRecord, without requiring the class in the classpath. See Accessing Domain Objects Without Domain Classes.