Service discovery in a microservice architecture with Pusher Channels

Introduction
Prerequisites
Pusher Channels registry
Directory structure
Building the NodeJS service
Building the Golang service
API gateway
Putting it all together
Conclusion

Introduction

In this tutorial, we will be implementing a service discovery using Pusher Channels. To demonstrate this, we will be building two services - both of them a unique ID generator. They will be implemented in both NodeJS and Golang. This is done so as to depict a real-life microservices scenario where services are built in multiple languages. In our use-case here, the NodeJS application will be version 1 while the second iteration will be in Golang. As said earlier, both of them are unique ID generators and basically all they do is generate a UUID. The Node service generates a UUID V5 while the Golang service will generate a UUID v4.

Microservices are an interesting pattern but they usually present a new class of problems. An interesting problem this pattern presents is communication between the bulk of services that make up the entire mesh. These services are run in multiple replicas - depending on scalability needs - and dynamic environments. Take Kubernetes as an example, deploying a container might run in server A, stopping and running it again might deploy it to server. For both of them, you’d obviously get an IP address but then again, you cannot guarantee it won’t change. At this rate, it is clear maintaining a hardcoded list of all service targets wouldn’t make any sense.

Even if we assume Kubernetes is not used and the application is being deployed pre Docker style - a single host, multiple instances. It is still hard to keep an updated list of all running instances. Just try to think of having to keep track of all ports you have assigned to an instance and the pain of having to update them manually.

To solve the communication problem within microservices, a pattern called service discovery emerged. With service discovery, each service when started would inform a central registry of its reachable IP address (8.8.8.8) and port (53). Also whenever it is being shut down, maybe as a result of CTRL+C, it deregisters itself from the registry. Popular implementations of service discovery include Consul and etcd.

We will also build another service - the API gateway - which will listen to connections from our unique ID generator services. This API gateway will be the public facing program as it will proxy request to an available instance of services registered with Pusher Channels.

Prerequisites

Node
Golang >= 1.10
Docker. We will be using this to run multiple copies of the services.
Create a free sandbox Pusher account or sign in.

Pusher Channels registry

Standards rule literally everything and we need to define the data structure our registry requires. Below is an example of what needs to be sent to the channel in order to successfully process its inclusion as a service backend.

1{
2      "prefix": "/v1", // App prefix to use for routing to this service
3      "port": 3000, // The port this service is running on
4      "address": "172.17.0.4", // The IP address where this service is reachable at.
5    }

Directory structure

Here is the directory structure you will need to create to follow this tutorial:

── $GOPATH/src/github.com/pusher-tutorial/pusher-channel-discovery ├── golang ├── nodejs ├── nodejs-api-gateway

Building the NodeJS service

This backend will make use of the Channels server SDK for Node. We will use it to send a register event to the central registry, in our case a channel.

To get started, create a folder called nodejs. This should match the directory structure above. In that project directory, you need to create another file called package.json. The package.json file should have the following contents:

1// pusher-channel-discovery/nodejs/package.json
2    {
3      "name": "nodejs-channel-discovery",
4      "version": "1.0.0",
5      "description": "Using pusher channel as a service registry",
6      "main": "index.js",
7      "dependencies": {
8        "body-parser": "^1.18.3",
9        "express": "^4.16.4",
10        "internal-ip": "^3.0.1",
11        "pusher": "^2.1.3",
12        "uuid": "^3.3.2"
13      },
14      "devDependencies": {
15        "nodemon": "^1.18.6"
16      }
17    }

Switch to your terminal app and navigate to the directory you created above - cd path/to/nodejs-channel-discovery. You then need to run the following command to install the dependencies needed to build this service.

    $ npm install

Next, create an index.js file in the root of the folder you created earlier and paste the contents below into it.

1// pusher-channel-discovery/nodejs/index.js
2    
3    const express = require('express');
4    const bodyParser = require('body-parser');
5    const os = require('os');
6    const uuidv5 = require('uuid/v5');
7    const uuidv4 = require('uuid/v4');
8    const Pusher = require('pusher');
9    const internalIp = require('internal-ip');
10    
11    const app = express();
12    const hostName = os.hostname();
13    const port = process.env.PORT || 3000;
14    
15    let pusher = new Pusher({
16      appId: process.env.PUSHER_APP_ID,
17      key: process.env.PUSHER_APP_KEY,
18      secret: process.env.PUSHER_APP_SECRET,
19      encrypted: process.env.PUSHER_APP_SECURE,
20      cluster: process.env.PUSHER_APP_CLUSTER,
21    });
22    
23    let svc = {};
24    
25    internalIp
26      .v4()
27      .then(ip => {
28        svc = {
29          prefix: '/v1',
30          port: port,
31          address: ip,
32        };
33    
34        console.log('Registering service');
35    
36        pusher.trigger('mapped-discovery', 'register', svc);
37      })
38      .catch(err => {
39        console.log(err);
40        process.exit();
41      });
42    
43    process.stdin.resume();
44    
45    process.on('SIGINT', () => {
46    
47      console.log('Deregistering service... ');
48    
49      // Send an exit signal on shutdown
50      pusher.trigger('mapped-discovery', 'exit', svc);
51    
52      // Timeout to make sure the signal sent to
53      // Pusher was successful before shutting down
54      setTimeout(() => {
55        process.exit();
56      }, 1000);
57    });
58    
59    app.use(bodyParser.json());
60    
61    app.use(function(req, res, next) {
62      // Uniquely identify the server that processed the request
63      res.header('X-Server', hostName);
64      next();
65    });
66    
67    app.get('/', function(req, res) {
68      res.status(200).send({ service: 'ID generator' });  
69    });
70    
71    app.get('/health', function(req, res) {
72      res.status(200).send({ status: 'ok' });
73    });
74    
75    app.post('/generate', function(req, res) {
76    
77      const identifier = req.body.id;
78      if (identifier === undefined) {
79        res.status(400).send({
80          message: 'Please provide an ID to use to generate your UUID V5',
81        });
82        return;
83      }
84    
85      if (identifier.length === 0) {
86        res.status(400).send({
87          message: 'Please provide an ID to use to generate your UUID V5',
88        });
89        return;
90      }
91    
92      res.status(200).send({
93        id: uuidv5(identifier, uuidv5.URL),
94        timestamp: new Date().getTime(),
95        message: 'UUID was successfully generated',
96      });
97    });
98    
99    app.listen(port, function() {
100      console.log(`Service is running at ${port} at ${hostName}`);
101    });

In the above code, three endpoints were created:

/ - This is the root handler of the application. This endpoint returns basic information of the service.
/health - This endpoint allows the application to notify others about its internal state.
/generate - This endpoint is meant for the generation of a UUID. It generates a version 5 UUID. This endpoint expects an ID from the caller which it then uses to compute the UUID.

Since we will be needing to run multiple copies of this service, let's set it up to run as a container. To do that, we create a Dockerfile. Create the file and paste the content below into the file.

1# pusher-channel-discovery/nodejs/Dockerfile
2    FROM node:10
3    COPY . ./
4    RUN npm install
5    CMD ["node", "."]

You need to save the file and build the image for this service. To do that, you should run the following command. Please note that this has to be done in the root of the nodejs directory.

    $ docker build -t pusher-channel-node .

You can then run the service. But to do that, you need to use the credentials from the Pusher Channels application you created at the start of the tutorial.

    $ docker run -p 127.0.0.1:3000:3000 -e PUSHER_APP_ID="XXXX" -e PUSHER_APP_KEY="XXXXXXX" -e PUSHER_APP_SECRET="XXXXXX" -e PUSHER_APP_CLUSTER="eu" -e PUSHER_APP_SECURE="1" pusher-channel-node

The above specifies that we want the service to be available on our machine at localhost:3000. You can test the service works as expected by trying to access http://localhost:3000/generate with a POST request that has the following as it's body {"id": "some random string"}. An example with curl is shown below:

    $ curl -d '{"id" : "3jhbj333"}' -H "Content-Type: application/json" -X POST http://localhost:3000/generate

Building the Golang service

We have decided to build another iteration of our Unique ID generator since the first version written in Node used UUID version 5 and we would prefer to use something much more random. UUID 5 is basically a way of hashing some value into 128 bits. So if you try generating multiple UUIDs with the same id value in your request, you keep on getting the same UUID. While that behavior is easy to change, let's assume we have tons of users and production code relying on that service already, we don’t want to change behavior but wouldn’t mind doing it right again. Hence the version two rewrite that uses UUID 4, which ensures complete randomness.

To start with, you need to create a directory that houses this service. You can go ahead to create one called golang. This directory needs to be created in a directory in accordance to the structure laid out at the beginning of the article .

You will also need to run go mod init in the newly created directory. This will make the project a Golang module.

The first set of actions we need to perform is connecting to Pusher Channels API, so we need a Go SDK. That can be done by the following command:

    $ go get github.com/pusher/pusher-http-go

We need to create a file called registry.go. Since we also have to properly structure the code into packages, it should be in a directory called registry. After which you should paste the following code into the registry.go file.

1// pusher-channel-discovery/golang/registry/registry.go
2    
3    package registry
4    
5    import (
6            "errors"
7            "net"
8    
9            pusher "github.com/pusher/pusher-http-go"
10    )
11    
12    type Event string
13    
14    const (
15            Register Event = "register"
16            Exit           = "exit"
17    )
18    
19    func (e Event) String() string {
20            switch e {
21            case Register:
22                    return string(Register)
23            case Exit:
24                    return string(Exit)
25            default:
26                    return ""
27            }
28    }
29    
30    const (
31            Channel = "mapped-discovery"
32    )
33    
34    type Registrar struct {
35            pusher *pusher.Client
36    }
37    
38    type Service struct {
39            // The path that is links to this service
40            Prefix string `json:"prefix"`
41    
42            // Public IP of the host running this service
43            Address net.IP `json:"address"`
44    
45            Port uint `json:"port"`
46    }
47    
48    func (s Service) Validate() error {
49            if s.Address == nil {
50                    return errors.New("addr is nil")
51            }
52    
53            if s.Port <= 0 {
54                    return errors.New("invalid HTTP port")
55            }
56    
57            return nil
58    }
59    
60    func New(client *pusher.Client) *Registrar {
61            return &Registrar{client}
62    }
63    
64    func (r *Registrar) do(svc Service, event Event) error {
65            if err := svc.Validate(); err != nil {
66                    return err
67            }
68    
69            _, err := r.pusher.Trigger(Channel, event.String(), svc)
70            return err
71    
72    }
73    
74    func (r *Registrar) Register(svc Service) error {
75            return r.do(svc, Register)
76    }
77    
78    func (r *Registrar) DeRegister(svc Service) error {
79            return r.do(svc, Exit)
80    }
81    
82    func (r *Registrar) IP() (net.IP, error) {
83            addrs, err := net.InterfaceAddrs()
84            if err != nil {
85                    return nil, err
86            }
87    
88            for _, addr := range addrs {
89                    if ipnet, ok := addr.(*net.IPNet); ok && ipnet.IP.IsGlobalUnicast() {
90                            if ipnet.IP.To4() != nil || ipnet.IP.To16() != nil {
91                                    return ipnet.IP, nil
92                            }
93                    }
94            }
95    
96            return nil, nil
97    }

That seems to be a lot, so here is a breakdown of what we have done above:

Defined multiple data types to conform to the channel registry structure defined above.
Implemented a Registrar type that triggers event to a channel. You can find those in the Register and DeRegister methods.

Our main goal is to generate unique IDs, we need to build an HTTP API that will allow for that.

We would be needing an HTTP router to help build our endpoints. For this, we would need a library called chi. To install it, run go get github.com/go-chi/chi. Since we would also be needing to generate unique IDs, it is safe to also install a UUID library. You will need to run go get github.com/google/uuid.

Create a folder called transport/web, and create an http.go file inside the newly created folder. Paste the code below in the http.go file:

1// pusher-channel-discovery/golang/transport/web/http.go
2    package web
3    
4    import (
5            "encoding/json"
6            "fmt"
7            "net/http"
8    
9            "github.com/go-chi/chi"
10            "github.com/google/uuid"
11    )
12    
13    type Server struct {
14            HostName string
15            Port     uint
16    }
17    
18    func Start(srv *Server) error {
19    
20            mux := chi.NewMux()
21    
22            mux.Use(func(next http.Handler) http.Handler {
23                    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
24                            w.Header().Set("Content-Type", "application/json")
25                            w.Header().Set("X-Server", srv.HostName)
26    
27                            next.ServeHTTP(w, r)
28                    })
29            })
30    
31            mux.Get("/health", func(w http.ResponseWriter, r *http.Request) {
32                    defer r.Body.Close()
33    
34                    w.WriteHeader(http.StatusOK)
35                    w.Write([]byte(`{ 'status' : 'OK' }`))
36            })
37    
38            mux.Post("/generate", func(w http.ResponseWriter, r *http.Request) {
39    
40                    defer r.Body.Close()
41    
42                    var response struct {
43                            Status int64     `json:"status"`
44                            ID     uuid.UUID `json:"id"`
45                    }
46    
47                    response.Status = 1
48                    response.ID = uuid.New()
49    
50                    w.WriteHeader(http.StatusOK)
51                    json.NewEncoder(w).Encode(&response)
52            })
53    
54            return http.ListenAndServe(fmt.Sprintf(":%d", srv.Port), mux)
55    }

In the code above, we have created two endpoints:

/health - This endpoint allows the application to notify others about its internal state.
/generate - This endpoint is responsible for creating the UUID. Unlike the first iteration written in NodeJS, it doesn't require any ID of any sort as it generates purely random UUIDs.

To tie up the registry and the HTTP API, we need to make our application able to run as a command line app. To do that, create a file called main.go in the cmd folder of the root application. You need to paste the following code in the main.go file:

1// pusher-channel-discovery/golang/cmd/main.go
2    package main
3    
4    import (
5            "errors"
6            "flag"
7            "fmt"
8            "log"
9            "os"
10            "os/signal"
11            "syscall"
12    
13            "github.com/adelowo/pusher-channel-discovery-go/registry"
14            "github.com/adelowo/pusher-channel-discovery-go/transport/web"
15            pusher "github.com/pusher/pusher-http-go"
16    )
17    
18    func main() {
19    
20            shutDownChan := make(chan os.Signal)
21            signal.Notify(shutDownChan, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
22    
23            port := flag.Uint("http.port", 3000, "Port to run HTTP server at")
24    
25            flag.Parse()
26    
27            appID := os.Getenv("PUSHER_APP_ID")
28            appKey := os.Getenv("PUSHER_APP_KEY")
29            appSecret := os.Getenv("PUSHER_APP_SECRET")
30            appCluster := os.Getenv("PUSHER_APP_CLUSTER")
31            appIsSecure := os.Getenv("PUSHER_APP_SECURE")
32    
33            var isSecure bool
34            if appIsSecure == "1" {
35                    isSecure = true
36            }
37    
38            client := &pusher.Client{
39                    AppId:   appID,
40                    Key:     appKey,
41                    Secret:  appSecret,
42                    Cluster: appCluster,
43                    Secure:  isSecure,
44            }
45    
46            reg := registry.New(client)
47    
48            ip, err := reg.IP()
49            if err != nil {
50                    log.Fatalf("could not fetch public IP address... %v", err)
51            }
52    
53            hostName, err := os.Hostname()
54            if err != nil {
55                    log.Fatalf("could not fetch host name... %v", err)
56            }
57    
58            svc := registry.Service{
59                    Prefix:  "/v2",
60                    Address: ip,
61                    Port:    *port,
62            }
63    
64            if err := reg.Register(svc); err != nil {
65                    log.Fatalf("Could not register service... %v", err)
66            }
67    
68            var errs = make(chan error, 3)
69            
70            go func() {
71                    srv := &web.Server{
72                            HostName: hostName,
73                            Port:     *port,
74                    }
75    
76                    errs <- web.Start(srv)
77            }()
78    
79            go func() {
80                    <-shutDownChan
81                    errs <- errors.New("Application is shutting down")
82            }()
83            
84            fmt.Println(<-errs)
85            reg.DeRegister(svc)
86    }

Since we need to run multiple copies of this service too, it would make total sense to run it in a container. To build a container image for this service, create a Dockerfile in the root of the project and paste the following code in it:

1## pusher-channel-discovery/golang/Dockerfile
2    FROM golang:1.11 as build-env
3    
4    WORKDIR /go/src/github.com/pusher-tutorial/pusher-channel-discovery-go
5    ADD . /go/src/github.com/pusher-tutorial/pusher-channel-discovery-go
6    
7    ENV GO111MODULE=on
8    
9    RUN go mod download
10    RUN go mod verify
11    RUN go install ./cmd
12    
13    ## A better scratch image
14    ## See https://github.com/GoogleContainerTools/distroless
15    FROM gcr.io/distroless/base
16    COPY --from=build-env /go/bin/cmd /
17    CMD ["/cmd"]

We need to build this image, so you need to run the following command in your terminal:

1$ export GO111MODULE=on
2    $ go mod tidy
3    $ docker build -t pusher-channel-go .

This will build the image and make it available to be run later on when we choose. Let's run it now:

    $ docker run -p 127.0.0.1:2000:3000 -e PUSHER_APP_ID="XXXX" -e PUSHER_APP_KEY="XXXXX" -e PUSHER_APP_SECRET="XXXX" -e PUSHER_APP_CLUSTER="eu" -e PUSHER_APP_SECURE="1" pusher-channel-go

You can verify the service is up and running by making a POST request to http://localhost:2000/generate. An example with curl is given below:

    $ curl -X POST localhost:2000/generate

API gateway

An API gateway is an application that provides a single entry point for a selected group of microservices. This plays hand in hand with service registration as it needs to be able to pull up information about those microservices so as to proxy requests to them. This part is called service discovery. You can read more about this pattern here.

We will implement this in NodeJS as we assume it already existed when we had only the first ID generator service. What we are building here is basically a reverse proxy. We subscribe to events published by the services we have built above and include the service in the routing table. Whenever we receive a request, we interpret the URL to determine what service is being requested. We use the prefix property defined above - in Pusher Channels registry - to achieve this.

Still in accordance with the directory structure laid out at the beginning of this tutorial, we need to create a new folder called nodejs-api-gateway. Inside that folder, we need a package.json file to define our dependencies. The following contents should be pasted inside the package.json file:

1// pusher-channels-discovery/nodejs-api-gateway/package.json
2    {
3      "name": "pusher-channels-api-gateway",
4      "version": "1.0.0",
5      "description": "",
6      "main": "index.js",
7      "dependencies": {
8        "pusher-js": "^4.3.1",
9        "redbird": "^0.8.0"
10      }
11    }

To install the dependencies declared above, you need to run:

    $ npm install

We then proceed to implement the proxy. We will use a node package called Redbird which is an awesome reverse proxy. You need to create an index.js file and it should have the following code as its contents:

1// pusher-channels-discovery/nodejs-api-gateway/index.js
2    
3    const port = process.env.PORT || 1500;
4    const domain = process.env.DOMAIN || `localhost:${port}`;
5    
6    const proxy = require('redbird')({ port: port });
7    const Pusher = require('pusher-js');
8    
9    const pusherSocket = new Pusher(process.env.PUSHER_APP_KEY, {
10      forceTLS: process.env.PUSHER_APP_SECURE === '1' ? true : false,
11      cluster: process.env.PUSHER_APP_CLUSTER,
12    });
13    
14    const channel = pusherSocket.subscribe('mapped-discovery');
15    
16    channel.bind('register', data => {
17      proxy.register(
18        `${domain}${data.prefix}`,
19        `http://${data.address}:${data.port}`
20      );
21    });
22    
23    channel.bind('exit', data => {
24      proxy.unregister(
25        `${domain}${data.prefix}`,
26        `http://${data.address}:${data.port}`
27      );
28    });

Here is a breakdown of what we have implemented above:

We define a domain constant. This defaults to localhost: PORT if the DOMAIN environmental value is not available.
We create a connection to Pusher channels and subscribe to the mapped-discovery channel.
We then bind a callback the register event. The callback picks out some critical data from the data it has received and uses that to update its routing table.
We also do the above for the exit channel. But in the instance, we remove the service from the routing table.

The reverse proxy supports load-balancing to multiple instances of our services as we will see shortly

As with other services we have built, this will also have Docker support. Create a Dockerfile and paste the following contents inside it:

1// pusher-channels-discovery/nodejs-api-gateway/Dockerfile
2    FROM node:10
3    COPY . ./
4    RUN npm install
5    CMD ["node", "."]

You also need to build the container image by running the following command in a terminal:

    $ docker build -t pusher-channel-api-gateway .

Putting it all together

You need to run the API gateway first before starting up every other service. State is not persisted and it will only listen to connections from microservices that are started after its last run. To start the API gateway container, run:

    $ docker run -p 127.0.0.1:1500:1500 -e PUSHER_APP_KEY="XXXXX" -e PUSHER_APP_SECURE="1" -e PUSHER_APP_CLUSTER="eu" pusher-channel-api-gateway

You then need to run the container for the service written in NodeJS by running the following command in your terminal:

    $ docker run -e PUSHER_APP_ID="XXXX" -e PUSHER_APP_KEY="XXXXXXX" -e PUSHER_APP_SECRET="XXXXXX" -e PUSHER_APP_CLUSTER="eu" -e PUSHER_APP_SECURE="1" pusher-channel-node

Keep an eye on the terminal window where you are running the API gateway, once the NodeJS service comes up, there should be information in the terminal. It should be in the following form:

    {"name":"redbird","hostname":"88c1cea2c10c","pid":1,"level":30,"from":{"protocol":"http:","slashes":true,"auth":null,"host":"localhost:1500","port":"1500","hostname":"localhost","hash":null,"search":null,"query":null,"pathname":"/v1","path":"/v1","href":"http://localhost:1500/v1"},"to":{"protocol":"http:","slashes":true,"auth":null,"host":"172.17.0.3:3000","port":"3000","hostname":"172.17.0.3","hash":null,"search":null,"query":null,"pathname":"/","path":"/","href":"http://172.17.0.3:3000/","useTargetHostHeader":false},"msg":"Registered a new route","time":"2018-11-11T20:08:02.632Z","v":0}

Shutting down the NodeJS service should produce something similar but with an Unregistered a route message.

You can access the Node service by sending a request to the API gateway, http://localhost:1500/v1/generate as previously shown. Why the v1? If you look at the part of the code that sends information to Pusher Channels, you will notice it has /v1 in its prefix property. If you take a look at the Golang implementation, you will find its prefix property with /v2, not /v1. What this means is all requests sent to the API gateway that starts from v1 would be proxied to the NodeJS service while those with v2 would be proxied to the Golang service. To run the Golang service, we need to run the following command:

    $ docker run -p 2000:3000 -e PUSHER_APP_ID="XXXX" -e PUSHER_APP_KEY="XXXXX" -e PUSHER_APP_SECRET="XXXX" -e PUSHER_APP_CLUSTER="eu" -e PUSHER_APP_SECURE="1" pusher-channel-go

All that is left for us to do now is to test that our requests are being proxied to the right services. I will be showing examples with curl.

1$ # for v1 
2    $ curl -i -d '{"id" : "3jhbj333"}' -H "Content-Type: application/json" -X POST http://localhost:1500/v1/generate
3    $ # for v2
4    $ curl -i -X POST localhost:1500/v2/generate

I have included the -i option as it will be useful for us to inspect the X-Server headers to also validate the request is being served by the right service. It will also be useful when we need to validate that requests are being proxied among multiple instances of a service. You need to run multiple instances of the NodeJS service, just open two or three terminal windows where you run the above docker command in each of them.

Once done, you should make requests to the API gateway and watch the value of X-Server change as requests are proxied in a round robin manner to all available instances.

    $ curl -i -d '{"id" : "3jhbj333"}' -H "Content-Type: application/json" -X POST http://localhost:1500/v1/generate

You can also try shutting down one or more instances of the available services and see what happens. Spoiler, requests are no longer proxied to them.

Conclusion

In this tutorial, we have leveraged Pusher Channels to implement service discovery and registration when dealing with microservices.

You can find the source code to this tutorial on GitHub.

Docs

Tutorials

Support

Glossary