In this article, I present a small Proof of Concept of communication between a browser (Js) and a server (Golang) using Protocol Buffers over Websockets. This idea emerged as part of a bigger personal project I’m working on and will probably show here in the future.

Objective of the project

The functionality of this proof of concept is pretty basic but it is a good showcase for quite a few technologies that are interesting to experiment with.
The idea is to create a persons database that is searchable and updatable from the browser.
For the frontend (browser), I will use plain javascript.
For the server, I will use Go.
For communication between the two, I will use WebSockets and the data will be serialized in the protocol buffers binary format.

Schema

Protobuf

If you are not familiar with Protocol Buffers, I’d recommend starting with the official page https://developers.google.com/protocol-buffers/.
In short, as the site says, “Protocol buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data”

One of the main advantages of protocol buffer binary format, over other ways of data serialization (E.g: XML, JSON), is that the messages are smaller and transmitted faster. Unlike XML, protocol buffer messages are not self-describing, so, when reading or writing, you will always need the message definition (the .proto file).
Protobuf are supported by every major language.

This is very nice, but how does it look like in this case? You can check it out here:
message.proto

Some decisions are worth mentioning. I define two message types that are used as events to trigger actions in the systems. Such as insert a Person or query Persons.

message UpsertPerson {
    Person person = 1;
}

message QueryPerson {
    string name_regexp = 1;
}

A trick I do to handle this events between client and server, is wrapping them into another generic message like this.

message ClientEvent {
    oneof event {
        UpsertPerson upsert_person = 1;
        QueryPerson query_person = 2;
    }
}

In this way, when I get a ClientEvent message in the server, I can easily figure out what is the actual event and deserialize correctly.

Then, I have he model itself with messages defined like:

message Person {
    string name = 1;
    repeated string characteristics = 2;
    float height = 3;
    int32 years = 4;
    repeated Address known_addresses = 5;
    bool has_police_record = 6;
    Gender gender = 7;
}

No big deal here. Just the Person entity with various fields to showcase different attribute types.

Another trick I’m using is defining a message called PersonDB like:

message PersonDB {
    repeated Person persons = 1;
}

I use this to serialize and deserialize the whole dataset from/to disk.

Going on, as mentioned earlier, the .proto file is just the definition of the messages I will be using. Now I need to generate actual code from this proto so my client and server application can understand. This means I will have to generate code in Golang and in Js.

Generating protobuf code for Golang

Download and install protobuf compiler from https://developers.google.com/protocol-buffers/docs/downloads.html
Install golang proto plugin doing

go get -u github.com/golang/protobuf/protoc-gen-go

Generate the source code for Go with the following command

protoc --proto_path=../ --go_out=. ../proto/message.proto

Generating protobuf code for Js

Install protoc (already have it from previous step)
Generate the source code for Js with the following command

protoc --proto_path=../ --js_out=library=MyMessages,binary:build/gen ../proto/message.proto

The server: Go

Let’s take a look at the main file that makes the server
https://github.com/miguelabate/js-go-proto-poc/blob/master/go-server-proto/main.go

To handle the websocket connections, I use the convenient Gorilla WebSockets library github.com/gorilla/websocket. This library handles the connection and the upgrade to WS.

E.g.:

Start the application with a basic http server listening on 8081 and set the ws function to handle the request

    var addr = flag.String("addr", ":8081", "http service address")
.
.
.
	http.HandleFunc("/ws", ws)
	log.Fatal(http.ListenAndServe(*addr, nil))

The start of the ws function that handles the request looks like:

func ws(w http.ResponseWriter, r *http.Request) {
	c, err := upgrader.Upgrade(w, r, nil)
	if err != nil {
		log.Print("upgrade:", err)
		return
	}
	defer c.Close()
.
.
.

As you can see, the first thing I do is upgrade the connection to WS and then I can easily use it for reading from the WS:

_, message, err := c.ReadMessage()

And writing:

err = c.WriteMessage(websocket.BinaryMessage, dataToSend)

A nice tip here is that the default WS upgrader won’t be able to handle Cross Origin connections, making it hard to do local development, so a trick here is to define your own upgrader like:

var upgrader = websocket.Upgrader{
	ReadBufferSize:  1024,
	WriteBufferSize: 1024,
	CheckOrigin: func(r *http.Request) bool { //doing this to allow cross origin for now
		return true
	},
}

That’s about it for the WebSocket connection handling. Now, other operations that I need to do in the server is deserialize the proto messages like:

clientEvent := &mymessages.ClientEvent{}
err = proto.Unmarshal(message, clientEvent)

Or serialize to send to the client:

result := new(mymessages.QueryPersonResult)
result.Persons = foundPersons

dataToSend, err := proto.Marshal(result)

Other thing I do in the server, is persisting the messages to disk:

persondb_bytes, _ := json.Marshal(personsdb_obj)
err := ioutil.WriteFile("myPersonsDB", persondb_bytes, 0644)

Then I can restore it between server restarts.

The client: Js

On the client side, I just have an html form with fields related to our Person entity and some buttons to do actions like open a WS connection, send a Person message to our persistence and query.

The main code for this can be seen here https://github.com/miguelabate/js-go-proto-poc/blob/master/js-client-proto/main.js

In our html we must include the necessary dependencies to work with js proto, and google clousures (the format used to generate the js sources based on the protos).

Can get these dependencies from:

include the protobuf js libs:
git@github.com:protocolbuffers/protobuf.git
https://github.com/protocolbuffers/protobuf/tree/master/js
get the closure library from google to be able to import the sources generated from the proto
https://github.com/google/closure-library

The includes should look like:

  <!-- closure lib from google -->
  <script src="libs-js/closure-library/closure/goog/base.js"></script>

  <!-- Protobuf -->
  <script src="libs-js/constants.js"></script>
  <script src="libs-js/utils.js"></script>
  <script src="libs-js/arith.js"></script>
  <script src="libs-js/decoder.js"></script>
  <script src="libs-js/encoder.js"></script>
  <script src="libs-js/reader.js"></script>
  <script src="libs-js/writer.js"></script>
  <script src="libs-js/map.js"></script>
  <script src="libs-js/message.js"></script>

  <!-- Generated from proto -->
  <script src="./build/gen/MyMessages.js"></script>

To connect to the WebSocket server I use the default HTML5 WebScoket capabilities.

In the following piece of JS code, it can be seen how to connect to the WS server and react to events, also it shows how to serialize and deserialize protobuf messages

    document.getElementById("open").onclick = function (evt) {
        if (ws) {
            return false;
        }
        ws = new WebSocket("ws://127.0.0.1:8081/ws");
        ws.binaryType = 'arraybuffer';
        ws.onopen = function (evt) {
            print("OPEN");
        }
        ws.onclose = function (evt) {
            print("CLOSE");
            ws = null;
        }
        ws.onmessage = function (evt) {
            let arrayBuffer = event.data;
            let message = proto.QueryPersonResult.deserializeBinary(arrayBuffer);

            console.log(message.toObject());
            let persons_retrieved = message.getPersonsList()
            print("QUERY RESULT TOTAL:" + persons_retrieved.length);
            for (let i = 0; i < persons_retrieved.length; i++) {
                print("QUERY RESULT:");
                print(prettyStringPerson(persons_retrieved[i]));
            }

        }
        ws.onerror = function (evt) {
            print("ERROR: " + evt.data);
        }
        return false;
    };

Another important piece of code is the one that sends (and serializes) to the WS server

    document.getElementById("send").onclick = function (evt) {
        if (!ws) {
            return false;
        }

        let message = new proto.ClientEvent();
        let upsertEvent = new proto.UpsertPerson();
        let newPerson = new proto.Person();
        newPerson.setName(inputName.value);
        newPerson.setCharacteristicsList(inputCharacteristics.value.split(";"));
        newPerson.setYears(inputYears.value);
        newPerson.setHeight(inputHeight.value);
        newPerson.setKnownAddressesList(getAddressesList(inputAddresses.value));
        newPerson.setGender(proto.Gender[inputGender.value]);
        newPerson.setHasPoliceRecord((inputPolRecord.value === 'true'));

        upsertEvent.setPerson(newPerson);
        message.setUpsertPerson(upsertEvent);
        print("SENT: " + message.toString());
        ws.send(message.serializeBinary(), { binary: true });
        return false;
    };

Conclusion

I have presented small POC showing how to set up a WebSocket server in GoLang and communicate with a browser JS client using protobuf messages. Interesting if you need to develop some kind of project (maybe a game?) that needs to keep a bidirectional connection between client-server.

Miguel Abate

Protobuf communication between Js-client and Go-server