How To Serialize Knowledge Successfully with Protocol Buffers – Grape Up

In a world of microservices, we regularly need to cross info between functions. We serialize knowledge right into a format that may be retrieved by either side. One of many serialization options is Protocol Buffers (Protobuf) – Google’s language-neutral mechanism. Messages might be interpreted by a receiver utilizing the identical or totally different language than a producer. Many languages are supported, comparable to Java, Go, Python, and C++.

An information construction is outlined utilizing impartial language via .proto information. The file is then compiled into code for use in functions. It’s designed for efficiency. Protocol Buffers encode knowledge into binary format, which reduces message measurement and improves transmission pace.

Defining Message Format

This .proto the file represents geolocation info for a given car.

1 syntax = "proto3";
3 bundle com.grapeup.geolocation;
5 import "google/kind/latlng.proto";
6 import "google/protobuf/timestamp.proto";
8 message Geolocation 
9  string vin = 1;
10  google.protobuf.Timestamp occurredOn = 2;
11  int32 pace = 3;
12  google.kind.LatLng coordinates = 4;
1 syntax = "proto3";

Syntax refers to Protobuf model, it may be proto2 or proto3.

1package com.grapeup.geolocation;

Package deal declaration prevents naming conflicts between totally different initiatives.

1 message Geolocation 
2  string vin = 1;
3  google.protobuf.Timestamp occurredOn = 2;
4  int32 pace = 3;
5  google.kind.LatLng coordinates = 4;

Message definition comprises a reputation and a set of typed fields. Easy knowledge varieties can be found, comparable to bool, int32, double, string, and so forth. It’s also possible to outline your personal varieties or import them.

1google.protobuf.Timestamp occurredOn = 2;

The = 1, = 2 markers determine the distinctive tag. Tags are a numeric illustration for the sphere and are used to determine the sphere within the message binary format. They need to be distinctive in a message and shouldn’t be modified as soon as the message is in use. If a discipline is faraway from a definition that’s already used, it should be reserved.

Area varieties

Except for scalar varieties, there are lots of different kind choices when defining messages. Listed below are few, however you could find all of them within the Language Information Language Guide (proto3)  |  Protocol Buffers  |  Google Developers .

Properly Recognized Varieties

1 import "google/kind/latlng.proto";
2 import "google/protobuf/timestamp.proto";
4 google.protobuf.Timestamp occurredOn = 2;
5 google.kind.LatLng coordinates = 4;

There are predefined varieties obtainable to make use of Overview  |  Protocol Buffers  |  Google Developers . They’re often known as Properly Know Varieties and need to be imported into .proto .

LatLng represents a latitude and longitude pair.

Timestamp is a selected cut-off date with nanosecond precision.

Customized varieties

1 message SingularSearchResponse 
2  Geolocation geolocation = 1;

You need to use your custom-defined kind as a discipline in one other message definition.


1 message SearchResponse 
2  repeated Geolocation geolocations = 1;

You may outline lists by utilizing repeated key phrase.


It will probably occur that in a message there’ll at all times be just one discipline set. On this case, TelemetryUpdate will comprise both geolocation, mileage, or gasoline degree info.

This may be achieved by utilizing oneof. Setting worth to one of many fields will clear all different fields outlined in oneof.

1 message TelemetryUpdate 
2  string vin = 1;
3  oneof replace 
4    Geolocation geolocation = 2;
5    Mileage mileage =3;
6    FuelLevel fuelLevel = 4;
10 message Geolocation 
11  ...
14 message Mileage 
15  ...
18 message FuelLevel 
19  ...

Take into accout backward-compatibility when eradicating fields. In the event you obtain a message with oneof that has been faraway from .proto definition, it won’t set any of the values. This habits is similar as not setting any worth within the first place.

You may carry out totally different actions based mostly on which worth is about utilizing the getUpdateCase() methodology.

1 public Optionally available<Object> getTelemetry(TelemetryUpdate telemetryUpdate) 
2        Optionally available<Object> telemetry = Optionally available.empty();
3        swap (telemetryUpdate.getUpdateCase()) 
4            case MILEAGE -> telemetry = Optionally available.of(telemetryUpdate.getMileage());
5            case FUELLEVEL -> telemetry = Optionally available.of(telemetryUpdate.getFuelLevel());
6            case GEOLOCATION -> telemetry = Optionally available.of(telemetryUpdate.getGeolocation());
7            case UPDATE_NOT_SET -> telemetry = Optionally available.empty();
9        return telemetry;

Default values

In proto3 format fields will at all times have a price. Due to this proto3 can have a smaller measurement as a result of fields with default values are omitted from payload. Nonetheless this causes one challenge – for scalar message fields, there isn’t any approach of telling if a discipline was explicitly set to the default worth or not set in any respect.

In our instance, pace is an non-obligatory discipline – some modules in a automotive may ship pace knowledge, and a few may not. If we don’t set pace, then the geolocation object could have pace with the default worth set to 0. This isn’t the identical as not having pace set on messages.

To be able to take care of default values you should utilize official wrapper varieties protobuf/wrappers.proto at main · protocolbuffers/protobuf . They permit distinguishing between absence and default. As an alternative of getting a easy kind, we use Int32Value, which is a wrapper for the int32 scalar kind.

1 import "google/protobuf/wrappers.proto";
3 message Geolocation 
4  google.protobuf.Int32Value pace = 3;

If we don’t present pace, it will likely be set to nil.

Configure with Gradle

When you’ve outlined your messages, you should utilize protoc, a protocol buffer compiler, to generate lessons in a selected language. The generated class can then be used to construct and retrieve messages.

To be able to compile into Java code, we have to add dependency and plugin in construct.gradle

1 plugins 
2    id '' model '0.8.18'
5 dependencies 
6    implementation ''

and setup the compiler. For Mac customers an osx particular model needs to be used.

1 protobuf 
2    protoc 
3        if (osdetector.os == "osx") 
4            artifact = "$protobuf_version:osx-x86_64"
5         else 
6            artifact = "$protobuf_version"

Code shall be generated utilizing generateProto job.

The code shall be positioned in construct/generated/supply/proto/most important/java in a bundle as laid out in .proto file.

We additionally want to inform gradle the place the generated code is positioned

1 sourceSets 
2    most important 
3        java 
4            srcDirs 'construct/generated/supply/proto/most important/grpc'
5            srcDirs 'construct/generated/supply/proto/most important/java'

The generated class comprises all the required strategies for constructing the message in addition to retrieving discipline values.

1 Geolocation geolocation = Geolocation.newBuilder()
2            .setCoordinates(LatLng.newBuilder().setLatitude(1.2).setLongitude(1.2).construct())
3            .setVin("1G2NF12FX2C129610")
4            .setOccurredOn(Timestamp.newBuilder().setSeconds(12349023).construct())
5            .construct();
7 LatLng coordinates = geolocation.getCoordinates();
8 String vin = geolocation.getVin();

Protocol Buffers – Abstract

As proven protocol buffers are simply configured. The mechanism is language agnostic, and it’s straightforward to share the identical .proto definition throughout totally different microservices.

Protobuf is well paired with gRPC, the place strategies might be outlined in .proto information and generated with gradle.

There’s official documentation obtainable Protocol Buffers  |  Google Developers and guides Language Guide (proto3)  |  Protocol Buffers  |  Google Developers .