Protobuf Protocol

Vison5/14/25About 4 minCore ConceptsprotobufSerialization

Introduction to Protobuf

Protocol Buffers (protobuf) is a language-neutral, platform-neutral, extensible mechanism developed by Google for serializing structured data, commonly used in scenarios such as communication protocols and data storage. Protobuf provides a compact binary format, which is highly efficient in multi-language, multi-platform environments, and is particularly common in distributed systems, microservices architectures, and RPC calls with high performance requirements.

Advantages of Protobuf
- Efficient: Compared to JSON and XML, Protobuf's binary format is more compact, occupying less bandwidth and storage space.
- Cross-Language Support: Protobuf supports multiple languages, including C++, Java, Go, Python, etc., making it suitable for multi-language projects.
- Version Compatibility: Protobuf message definitions support backward compatibility, facilitating updates to message formats without affecting existing clients.

Protobuf3 Basic Syntax

Core Concepts
- message: Similar to classes or structs, defines the structure of data.
- field: Each field in a message has a unique identifier (tag), used for locating during serialization.
- service: Used to define RPC services, proto3 supports integration with gRPC.
- import: Used to import other .proto files, resolving dependencies.
- option: Used to set compiler options, such as the generated Go package name.
Basic Data Types
- int32, int64: Signed integers
- uint32, uint64: Unsigned integers
- bool: Boolean type
- string: String type
- bytes: Byte type
- float, double: Floating-point types
- enum: Enum type

An Example Protobuf File user.proto

syntax = "proto3";

package user;

option go_package = "user";

service UserService {
  rpc GetUser (GetUserRequest) returns (User);
}

message GetUserRequest {
  int32 id = 1;
}

// Define the message for user information
message User {
  int32 id = 1;
  string name = 2;
  string email = 3;
  repeated PhoneNumber phones = 4;
}

// Define phone type
message PhoneNumber {
  string number = 1;
  PhoneType type = 2;
}

// Enumerate phone types
enum PhoneType {
  MOBILE = 0;
  HOME = 1;
  WORK = 2;
}

Protobuf uses .proto files to define data structures and RPC services, which then need to be compiled into target language code using the protoc compiler. You can download and install the protoc compiler for various platforms from the Protobuf Official GitHub Repository.

Protobuf Specifications in sponge

Sponge has a built-in plugin based on protobuf3 for generating http/gRPC API interface code, which can be flexibly adjusted according to business needs.

In services created by sponge, proto files are uniformly stored in the api directory. The directory path is: api/<service name>/<version>/proto_file, for example: api/user/v1/user.proto.

.
├── api                        # Directory for storing proto files
│   └──  user                  # Service name
│        └── v1                # Version number
│            └── user.proto    # Proto file
└── third_party                # Third-party imported proto files

Taking a user service as an example, with the go module name edusys, the content of the user.proto file is as follows:

syntax = "proto3";

package api.user.v1;

option go_package = "edusys/api/user/v1;v1";

import "api/user/types.proto";
import "validate/validate.proto";

service UserService {
  rpc List(ListUserRequest) returns (ListUserReply) {};
}

message ListUserRequest {
  api.user.v1.Params params = 1 [(validate.rules).message.required = true];
}

message ListUserReply {
  int64 total = 1;
  repeated User users = 2;
}

message User {
  int32 id = 1;
  string name = 2;
  string email = 3;
}

package Specification

The definition of package needs to correspond to the path of the api directory. The naming format is api.<service name>.<version>, for example api.user.v1.

go_package Specification

The definition of go_package also needs to correspond to the path of the api directory. The naming format is <go module name>/api/<service name>/<version>;<version>, for example edusys/api/user/v1;v1.

import Specification

To depend on other proto files within this service, find the dependent proto file (e.g., types.proto), check its package name (e.g., api.user.v1), and then import "api/user/types.proto"; in the current proto file.

To depend on third-party proto files under the third_party directory, find the dependent proto file (e.g., annotations.proto), check its package name (e.g., google.api), and then import "google/api/annotations.proto"; in the current proto file.

message Specification

If the proto file is only used for gRPC services, only pay attention to specification points 1, 2, and 3 below.
If the proto file is used for Web API interfaces, then pay attention to specification points 1, 2, 3, 4, 5, and 6 below.

It is recommended to use camelCase for message field names, such as firstName, lastName, etc.
If a message field name ends with "id", it is recommended to use the xxxID naming format, such as userID, orderID, etc.
Add validation rules [https://github.com/envoyproxy/protoc-gen-validate#constraint-rules], for example:
```
uint64 id = 1 [(validate.rules).uint64.gte = 1];
```
If the route includes path parameters, such as /api/v1/userExample/{id}, the defined message must include the name of the path parameter, and this name should add a new tag, for example:
```
int64 id = 1 [(tagger.tags) = "uri:\"id\""];
```
If the request URL is followed by query parameters, such as /api/v1/getUserExample?name=Tom, a form tag must be added when defining query parameters in the message, for example:
```
string name = 1 [(tagger.tags) = "form:\"name\""];
```
If a message field name contains an underscore (e.g., "field_name"), it will cause the JSON field name of Swagger request parameters to differ from the GRPC JSON tag name. There are two solutions for this:
- Solution 1: Remove the underscore from the message field name.
- Solution 2: Use the third-party tool protoc-go-inject-tag to modify the JSON tag name, for example:
```
string first_name = 1; // @gotags: json:"firstName"
```

Custom Specification for google.api.http selector field

The selector field was originally used to set the mapping relationship between HTTP methods and RPC methods. However, sponge does not need this mapping relationship, so we extend it to define the generation of corresponding gin-related code. The extended values are [ctx] or [no_bind].

[ctx]: Indicates that this route passes gin.Context through to the handler, and the ShouldBinding parameter will still be executed internally. Proto file example:

    // Login
    rpc Login(LoginRequest) returns (LoginReply) {
        option (google.api.http) = {
            post: "/api/v1/auth/login"
            body: "*"
            selector: "[ctx]"
        };
    }

Example of getting gin.Context in the handler:

    // Login Login
    func (h *userHandler) Login(ctx context.Context, req *userV1.LoginRequest) (*userV1.LoginReply, error) {
        c, ctx := middleware.AdaptCtx(ctx) // Variable c is gin.Context

        // ......
    }

[no_bind]: Indicates that this route is set to pass gin.Context through and skips the ShouldBinding parameter. Proto file example:

    // Upload file
    rpc Upload(UploadRequest) returns (UploadReply) {
        option (google.api.http) = {
            post: "/api/v1/upload"
            body: "*"
            selector: "[no_bind]"
        };
    }

Example of getting gin.Context in the handler:

    // Upload Upload file
    func (h *userHandler) Upload(ctx context.Context, req *userV1.UploadRequest) (*userV1.UploadReply, error) {
        c, ctx := middleware.AdaptCtx(ctx) // Variable c is gin.Context

        // Handle file upload logic
    }

Sponge combines the advantages of protobuf3 and is used in Web, RPC systems, microservices, and distributed systems. By defining simple .proto files, developers can automatically generate serialization and deserialization code, greatly reducing the amount of manual coding work and improving code performance and maintainability.

References:

Protocol Buffers 3 Guides