How to use Protobuf with Go?
Have you ever been in a situation where you do not know the structure of data that services are either consuming or publishing?
Did you know that you have to write a schema only once which defines the structure data and the same schema can be used as validation before sending & receiving data using Protocol Buffers?
In this tutorial, you will learn about using Protobuf. You will send & receive a message asynchronously using Protobuf with RabbitMQ.
Table Of Contents
- Prerequisites
- Protobuf
- Bootstrapping Project
- Creating Protobuf Messages
- Publishing a Message
- Consuming a Message
- Conclusion
Prerequisites
This tutorial will be a hands-on demonstration. If you’d like to follow along, be sure you have the following:
- You have running Docker Composer in your machine.
- You have a basic understanding of Docker, GO, Rabbitmq.
Protobuf
Protobufs are the Messaging Contracts. They are schema that describes the structure of the message. These files are written in .proto
extension which is language-neutral i.e write once & use anywhere. Proto Messages are the structured schema that is inside .proto
files
You will write Proto Messages and with the compiler called Proto Compiler, you can generate the code which will have a way to serialize & deserialize data into that format.
There are many benefits of using Messaging Contracts like Protobuf :
- With these Proto Messages, you will know what type of data you are sending & receiving
- Serves as a documentation
- Supports validation for serializing & deserializing
- Proto messages are language-neutral and it supports various programming languages which means you can publish messages from Go & start consuming them from Php / Node etc
Bootstrapping Project
For creating a new test project follow the steps below:
- Copy
Dockerfile
&docker-compose.yaml
from this repository which will dockerized Go source code, Proto compiler & RabbitMQ - Create a file
publish.go
where you will write some code to publish events - Create a file
consume.go
where you will write some code to consume events
Creating Protobuf Messages
In an application, while you sending messages you will serialize data using the definition from Proto Messages. And when consuming messages you will deserialize them using the same Proto Messages. Let's create a Proto Message so your services can use them to send and receive messages.
- Create a folder
messages
and inside that folder create a file calleduser_registered.go
- Inside
user_registered.go
copy the following codes. Let's understand the code
syntax
defines which proto version you are going to usego_package
defines the path where all your generated Protobuf code will be into`import “validate/validate.proto”;
it is importing the validator so that you can validate message using various rules without writing it by yourself- All message starts with a message keyword followed by name of the message. Inside the block, you can write key-value pairs where the key defines name & type & the value defines its order & rules if there are any. In our case, you are defining the first field
userId
which is an integer and should be greater than 0 and the next is an email which is a string and should not be empty.
3. Run make build-proto
on your project root. This command will generate two files user_registered.pb.go and user_registered.pb.validate.go
that define type & includes all the validation logic that you specified on the text which you can later use on your code while serializing & deserializing.
Publishing a Message
As you have successfully created a Proto Message. Let's use the same auto-generated code while sending messages to other services for this you need to write a way to publish messages. Follow the following steps:
- Create a directory
pkg
and a filerabbitmq.go
inside pkg dir where you will make a connection with RabbitMQ. The following code the functiongetConnection
is importing library and making a connection with RabbitMQ server and upon error functionfailOnError
is terminating the application with an error.
2. After successfully making a connection, write a function that will publish a message to RabbitMQ. The following code accepts queueName
and payload as a message
which will write a message to default exchange to JSON format.
3. Create publish.go
which will import publish function from rabbitmq.go
to send data. The following code is using a struct that was generated by the proto using the proto compiler and it is encoding them into JSON after that it is calling to publish the function that you created earlier to send data to UserRegistered
queue.
Try running make publish
to verify that your code is actually working if you see Data has been published
the output that means you were successful in publishing a message.
Consuming a Message
Until now you defined a photo message then you created a rabbitmq.go
to create a way to publish messages & you used the same function publish.go
to publish messages to RabbitMq. Now let's consume the message that you published earlier.
- Inside
rabbit. go
add a function that will help in consuming a message. The following code makes a connection to RabbitMq & returns a channel that you can leverage to listen to incoming messages.
2. Inside consume.go
listen to all the messages and then print them. Let's understand the following code :
- The main function is listening to message by creating a consuming client. While deserializing message it is importing
UserRegistered
struct from the auto-generated code to define the type of data. There might be cases where the producer might send invalid data due to various reasons so it is better to validate that data you are consuming are as per the contracts that you specified. validateMessage
accepts the JSON data and tries to validate as per theuser_registered.proto
definition and if the data breaks any validation rules it will throw a validation error
Conclusion
Throughout this tutorial, you have learned Protobuf & how to use them with GO & RabbitMQ.
Protobuf is one of the ways to implement Messaging Contracts there are more options like Pact, AsyncApi
Now with this knowledge, you can use Protobuf on any programming language to send & receive the data. You have also learned that it was easier to predict the data that you are consuming or publishing.
Let me know if you have any comments /suggestions/feedback.
Source code: https://github.com/ujwaldhakal/golang-protobuf
Ref: https://stackshare.io/protobuf