How to use Protobuf with Go?
Have you ever been in a situation where you do not know the structure of data that services are either consuming or publishing?
Did you know that you have to write a schema only once which defines the structure data and the same schema can be used as validation before sending & receiving data using Protocol Buffers?
In this tutorial, you will learn about using Protobuf. You will send & receive a message asynchronously using Protobuf with RabbitMQ.
Table Of Contents
- Bootstrapping Project
- Creating Protobuf Messages
- Publishing a Message
- Consuming a Message
This tutorial will be a hands-on demonstration. If you’d like to follow along, be sure you have the following:
- You have running Docker Composer in your machine.
- You have a basic understanding of Docker, GO, Rabbitmq.
Protobufs are the Messaging Contracts. They are schema that describes the structure of the message. These files are written in
.proto extension which is language-neutral i.e write once & use anywhere. Proto Messages are the structured schema that is inside
You will write Proto Messages and with the compiler called Proto Compiler, you can generate the code which will have a way to serialize & deserialize data into that format.
There are many benefits of using Messaging Contracts like Protobuf :
- With these Proto Messages, you will know what type of data you are sending & receiving
- Serves as a documentation
- Supports validation for serializing & deserializing
- Proto messages are language-neutral and it supports various programming languages which means you can publish messages from Go & start consuming them from Php / Node etc
For creating a new test project follow the steps below:
docker-compose.yamlfrom this repository which will dockerized Go source code, Proto compiler & RabbitMQ
- Create a file
publish.gowhere you will write some code to publish events
- Create a file
consume.gowhere you will write some code to consume events
Creating Protobuf Messages
In an application, while you sending messages you will serialize data using the definition from Proto Messages. And when consuming messages you will deserialize them using the same Proto Messages. Let's create a Proto Message so your services can use them to send and receive messages.
- Create a folder
messagesand inside that folder create a file called
user_registered.gocopy the following codes. Let's understand the code
syntaxdefines which proto version you are going to use
go_packagedefines the path where all your generated Protobuf code will be into
`import “validate/validate.proto”;it is importing the validator so that you can validate message using various rules without writing it by yourself
- All message starts with a message keyword followed by name of the message. Inside the block, you can write key-value pairs where the key defines name & type & the value defines its order & rules if there are any. In our case, you are defining the first field
userIdwhich is an integer and should be greater than 0 and the next is an email which is a string and should not be empty.
make build-proto on your project root. This command will generate two files
user_registered.pb.go and user_registered.pb.validate.go that define type & includes all the validation logic that you specified on the text which you can later use on your code while serializing & deserializing.
Publishing a Message
As you have successfully created a Proto Message. Let's use the same auto-generated code while sending messages to other services for this you need to write a way to publish messages. Follow the following steps:
- Create a directory
pkgand a file
rabbitmq.goinside pkg dir where you will make a connection with RabbitMQ. The following code the function
getConnectionis importing library and making a connection with RabbitMQ server and upon error function
failOnErroris terminating the application with an error.
2. After successfully making a connection, write a function that will publish a message to RabbitMQ. The following code accepts
queueName and payload as a
message which will write a message to default exchange to JSON format.
publish.go which will import publish function from
rabbitmq.go to send data. The following code is using a struct that was generated by the proto using the proto compiler and it is encoding them into JSON after that it is calling to publish the function that you created earlier to send data to
make publish to verify that your code is actually working if you see
Data has been published the output that means you were successful in publishing a message.
Consuming a Message
Until now you defined a photo message then you created a
rabbitmq.go to create a way to publish messages & you used the same function
publish.go to publish messages to RabbitMq. Now let's consume the message that you published earlier.
rabbit. goadd a function that will help in consuming a message. The following code makes a connection to RabbitMq & returns a channel that you can leverage to listen to incoming messages.
consume.go listen to all the messages and then print them. Let's understand the following code :
- The main function is listening to message by creating a consuming client. While deserializing message it is importing
UserRegisteredstruct from the auto-generated code to define the type of data. There might be cases where the producer might send invalid data due to various reasons so it is better to validate that data you are consuming are as per the contracts that you specified.
validateMessageaccepts the JSON data and tries to validate as per the
user_registered.protodefinition and if the data breaks any validation rules it will throw a validation error
Throughout this tutorial, you have learned Protobuf & how to use them with GO & RabbitMQ.
Protobuf is one of the ways to implement Messaging Contracts there are more options like Pact, AsyncApi
Now with this knowledge, you can use Protobuf on any programming language to send & receive the data. You have also learned that it was easier to predict the data that you are consuming or publishing.
Let me know if you have any comments /suggestions/feedback.
Source code: https://github.com/ujwaldhakal/golang-protobuf