Creating Web Server Based on Protobuf

Overview

The Create web server based on protobuf solution provides a complete set of solutions for general web backend development. This solution allows developers to focus solely on implementing business logic, while other basic code (such as framework code, interface definitions, etc.) is automatically generated by sponge. It's worth noting that business logic code can also be generated with the help of sponge's built-in AI assistant, further reducing manual coding effort.

Note

Unlike another Web development solution, Create web server based on sql (see Based on SQL), this solution supports automatic generation of custom API code through Protobuf. This is the core difference between the two. From a functional scope perspective, Create web server based on sql can be considered a subset of Create web server based on protobuf.

Applicable Scenarios: General Web project backend development, especially complex business scenarios requiring flexible API interface definitions.

Create web server based on protobuf defaults to no ORM components. It supports development using built-in ORM components or custom ORM components. Below, we will take the built-in ORM component gorm as an example to detail the steps for developing a web service.

Tips

Built-in ORM components: gorm, mongo. Supports database types: mysql, mongodb, postgresql, sqlite.

Custom ORM components: For example, sqlx, xorm, ent, etc. When choosing a custom ORM component, developers need to implement the relevant code themselves. Click to view the section: Explanation of Using Custom ORM Components.

Preparation

Environment Requirements:

• sponge installed
• mysql database service
• Database table structure
• Proto file, for example user.proto.

Note

Generating handler CRUD code requires a mysql service and database tables. You can start the mysql service using the docker script and then import the example SQL.

Create Web Server

Execute the command sponge run in the terminal to enter the code generation UI:

1. Click the left menu bar [Protobuf] → [Create Web Server];
2. Select the proto file(s) (multiple selection is allowed);
3. Then fill in other parameters. Hover the mouse over the question mark ? to view parameter descriptions.

Tips

If the Large Repository Type option is enabled in the parameters, all subsequent related code generation must maintain this setting.

After filling in the parameters, click the Download Code button to generate the web service project code, as shown in the figure below:

web-http-pb

Equivalent Command

sponge web http-pb --module-name=user --server-name=user --project-name=edusys --protobuf-file=./user.proto

Tips

• The generated Web service code directory is named by default in the format service name-type-time. You can modify the directory name as needed.

• The system automatically saves the records of successfully generated code, making it convenient for the next time code is generated. The previous partial parameters will be displayed when the page is refreshed or reopened.

Directory Structure

The generated code directory structure is as follows:

.
├─ api
│   └─ user
│       └─ v1
├─ cmd
│   └─ user
│       ├─ initial
│       └─ main.go
├─ configs
├─ deployments
│   ├─ binary
│   ├─ docker-compose
│   └─ kubernetes
├─ docs
├─ internal
│   ├─ config
│   ├─ ecode
│   ├─ handler
│   ├─ routers
│   └─ server
├─ scripts
└─ third_party

Code Structure Diagram

The created web service code adopts the classic "Egg Model" architecture:

web-http-pb-anatomy

Code Call Chain Description

The Web service code generated by sponge uses a layered architecture. The complete call chain is as follows:

cmd/user/main.go → internal/server/http.go → internal/routers/router.go → internal/handler → internal/dao → internal/model

The handler layer is mainly responsible for API processing. If more complex business logic needs to be handled, it is recommended to add an additional business logic layer (such as internal/biz) between the handler and dao. For details, please click to view the Code Layered Architecture section.

Test Web Service API

Unzip the code file, open the terminal, switch to the web service code directory, and execute the command:

# Generate and merge api related code
make proto

# Compile and run the service
make run

make proto command explained

1. Usage Suggestion
   Only execute this command when the API description in the proto file changes. Otherwise, skip this command and directly run make run.

2. This command performs the following automated operations in the background

   • Generate *.pb.go files
   • Generate router registration code
   • Generate error code definitions
   • Generate Swagger documentation
   • Generate API template code
   • Automatically merge API template code

3. Safety Mechanism

• Original business logic is preserved during code merging.
• Code is automatically backed up before each merge to:
  • Linux/Mac: /tmp/sponge_merge_backup_code
  • Windows: C:\Users\[username]\AppData\Local\Temp\sponge_merge_backup_code

Open http://localhost:8080/apis/swagger/index.html in your browser to view the API documentation on the page, as shown in the figure below:

Note

Before implementing the business logic, Swagger requests will return a 500 error. This is because the generated template code (internal/handler/xxx.go) contains panic("implement me") in each method, prompting the developer to implement the specific business logic themselves or use the built-in AI assistant.

web-http-swagger

Swagger Configuration Explanation

If you modified the HTTP port in the configs/service name.yml configuration file (e.g., changed from 8080 to 9090), you must perform the following operations synchronously:

1. Modify the @host value in the code file cmd/service name/main.go to the new port (e.g., localhost:9090)
2. Re-execute make docs to generate the documentation

Otherwise, API requests will fail due to port inconsistency.

Add CRUD API

Standardized CRUD APIs are often needed in projects. The CRUD APIs generated by sponge can be seamlessly added to the web service code without writing any Go code.

Generate Handler CRUD Code

1. Click the left menu bar [Public] → [Generate Handler CRUD Code];
2. Select the database mysql, fill in the database dsn, then click the Get Table Names button, and select the mysql tables (multiple selection is allowed);
3. Enable protobuf type;
4. Fill in parameters like module name, service name, etc.

After filling in the parameters, click the Download Code button to generate the handler CRUD code, as shown in the figure below:

web-http-handler-pb

Equivalent Command

# Full command
sponge web handler-pb --module-name=user --db-driver=mysql --db-dsn="root:123456@(192.168.3.37:3306)/school" --db-table=course,teach

# Simplified command (use --out to specify the service code directory, generated code is automatically merged into the specified service directory)
sponge web handler --db-driver=mysql --db-dsn="root:123456@(192.168.3.37:3306)/school" --db-table=course,teach --out=user

Code Directory Structure

The generated CRUD handler code directory is as follows:

.
├─ api
│   └─ user
│       └─ v1
└─ internal
    ├─ cache
    ├─ dao
    ├─ ecode
    ├─ handler
    └─ model

Test CRUD API

Perform simple configuration before starting the service (only needs to be done once):

• Modify the MySQL address in configs/service name.yml
• Uncomment the MySQL and cache initialization code in cmd/service name/initial/initApp.go
• Uncomment the relevant resource release code in registerClose.go

Unzip the code, move the internal and api directories to the web service code directory, and execute the command in the terminal:

# Generate and merge api related code
make proto

# Compile and run the service
make run

Note

If you encounter make proto errors, please refer to the Frequently Asked Questions section.

Refresh the Swagger page http://localhost:8080/apis/swagger/index.html to view and test the newly added CRUD APIs.

Tips

The List interface in CRUD API supports powerful custom condition pagination query functionality. Click to view detailed usage rules: Custom Condition Pagination Query.

Develop Custom API

As business requirements grow, developers may need to add new custom APIs. sponge adopts a "define-to-generate" development model, allowing for rapid custom API development, mainly divided into the following three steps:

1. Define API
   Declare the request/response format of the API in the .proto file.

2. Implement Logic
   Fill in the core business logic code in the automatically generated code templates.

3. Test Verification
   Test the API in the built-in Swagger without relying on third-party tools like Postman.

Below, we will take adding a "Change Password" API as an example to detail the development process.

1. Define API

Go to the project directory api/user/v1, edit the user.proto file, and add the description information for the Change Password API:

import "validate/validate.proto";
import "tagger/tagger.proto";

service user {
  // ...

  // Change password, describe the specific implementation logic here, telling sponge's built-in AI assistant how to generate the business logic code
  rpc ChangePassword(ChangePasswordRequest) returns (ChangePasswordReply) {
    option (google.api.http) = {
      post: "/api/v1/user/change_password"
      body: "*"
    };
    option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_operation) = {
      summary: "change password",
    };
  }
}

message ChangePasswordRequest {
  uint64 id = 1 [(validate.rules).uint64.gte  = 1, (tagger.tags) = "uri:\"id\"" ];
  string password = 2 [(validate.rules).string.min_len = 6];
}

message ChangePasswordReply {
}

After adding the API description information, execute the command in the terminal:

# Generate and merge api related code
make proto

2. Implement Business Logic

There are two ways to implement business logic code:

• Manually write business logic code

  Open the code file internal/handler/user.go and write the business logic code in the ChangePassword method function, referring to the template code.

  Note

  In developing custom APIs, you may need to perform database CRUD or cache operations. You can reuse the following code:

  • Database CRUD operations
  • Cache operations

• Automatically generate business logic code

  Sponge provides a built-in AI assistant to generate business logic code. Click to view the section on AI Assistant generates code.

  The business logic code generated by the AI assistant may not fully meet the actual requirements and needs to be modified according to the actual situation.

3. Test Custom API

After implementing the business logic code, execute the command in the terminal:

# Compile and run the service
make run

Refresh the Swagger interface http://localhost:8080/apis/swagger/index.html in your browser and test the custom API on the page.

Service Configuration Description

The Create web server based on protobuf solution offers a wealth of configurable components. You can flexibly manage these components by modifying the configs/service name.yml configuration file.

Component Management Explanation

Custom gin middleware:

• You can add or replace middleware in internal/routers/routers.go.
• If API authentication is required, add middleware.Auth() in internal/routers/table name.go.

Default Enabled Components

Component	Function Description	Configuration Documentation
logger	Logging component • Default terminal output • Supports console/json format • Supports log file splitting and retention	Logger Configuration
enableMetrics	Prometheus metrics collection • Default route /metrics	Monitoring Configuration
enableStat	Resource monitoring • Records CPU/memory usage per minute • Automatically saves profile if threshold is exceeded	Resource Statistics

Default Disabled Components

Component	Function Description	Configuration Documentation
cacheType	Cache support (Redis/Memory)	Redis Configuration
enableHTTPProfile	Performance analysis (pprof) • Default route /debug/pprof/	-
enableLimit	Adaptive request limiting	Rate Limiting Configuration
enableCircuitBreaker	Circuit breaker protection	Circuit Breaker Configuration
enableTrace	Distributed tracing	Tracing Configuration
registryDiscoveryType	Service registration and discovery • Consul/Etcd/Nacos	Service Registration and Discovery Configuration
database	Database support • MySQL/MongoDB/PostgreSQL/SQLite	Gorm Configuration MongoDB Configuration

Configuration Update Process

If you add or change field names in the configuration file configs/service name.yml, you need to update the corresponding Go code by executing the command in the terminal:

# Regenerate configuration code
make config

Note

If you only modify the field values in the configuration file, you do not need to execute the make config command; just recompile and run the service.

Tips

To learn more about the components and configurations, click to view the Components and Configuration section.