Work with Protobuf

Flow supports reading Protobuf messages, either as a message format for HTTP endpoints or from a streaming query originating from Kafka.

You can annotate Protobuf sources directly with Taxi metadata, or you can import through the Schema Importer.

Embed semantic types in Protobuf

Within a Protobuf, fields can be annotated with Taxi metadata to indicate the semantic type used on a field.

Fields are annotated using the taxi.dataType extension (which must be imported in the Protobuf file).

An example is shown:

import "taxi/dataType.proto";

message CafeDrink {
  optional string customer_name = 1 [(taxi.dataType)="foo.CustomerName"];
  optional int32 customer_id = 2 [(taxi.dataType)="foo.CustomerId"];

  enum Foam {
    NOT_FOAMY_AND_QUITE_BORING = 1;
    ZOMG_SO_FOAMY = 3;
  }
}

Datatype proto extension

Protobufs do not currently support importing external type definitions, so you need to include the source of the dataType proto definition directly in your Protobuf project.

By convention, the Protobuf definition should be written to taxi/dataType.proto within your Protobuf project.

import "google/protobuf/descriptor.proto";

package taxi;

extend google.protobuf.FieldOptions {
  optional string dataType = 50002;
}

Generated Taxi

When a Protobuf with Taxi metadata is imported by Flow, a Taxi representation is automatically generated.

You generally wouldn’t hand-craft a Protobuf Taxi declaration. Taxi has tooling to generate this automatically. The source is shown here for reference only.

@lang.taxi.formats.ProtobufMessage(packageName = "" , messageName = "CafeDrink")
model CafeDrink {
   @lang.taxi.formats.ProtobufField(tag = 1 , protoType = "string")
   customer_name : foo.CustomerName?

   @lang.taxi.formats.ProtobufField(tag = 2 , protoType = "int32")
   customer_id : foo.CustomerId?
}

Import a Protobuf through the UI

You can import a Protobuf schema directly through the UI.

Import the Protobuf file

Click on Schema Explorer, then Add new
Select Protobuf from the dropdown list

protobuf drop down flow

To provide the Protobuf, you can:

Upload a single Protobuf schema
Upload a zip file containing Protobuf schemas
Specify a URL to a single Protobuf file
Specify a URL to a zip file of Protobuf schemas

Then click Create.

Preview the generated types

A preview is shown, containing the models, fields and types that were imported from the Protobuf schema.

schema protobuf preview flow

You can change any of the types (e.g., swapping a primitive type with a more specific semantic type), by clicking on the pencil icon next to the type name, and searching for the desired type.

Once you’re satisfied, click Save, and the schema will be updated.