Pusher Channels as an alternative messaging queue

Introduction

In this tutorial, we will be building a message queue backed up by Pusher Channels. The application we will build will be a typical login service which upon a successful authentication, an email is sent to the authenticated user informing him of the authentication process and where it originated from. This is quite common with web applications - Twitter, GitHub and Slack do this all the time. We will build the login service in Golang while the email service will be written in NodeJS. The Golang application will publish the data to Pusher channels while the Node.js service will be subscribe to the particular channel and send the email to the user.

Messaging queues are an interesting technique used to improve scalability and a bit of abstraction between the producer and the receiver/consumer as they don’t have to be connected in whatever form. A message queue is nothing much more than a list of messages being sent between two or more applications. A message is basically data produced by an application usually called the producer. That data is then sent into the queue to be picked up by another totally different application - known as the consumer.

Prerequisites

• Golang ( >= 1.9)
• Node.js ( >= 7 )
• Free sandbox Pusher account or sign in.

Building the login service

Let’s set up a simple login Golang service. Due to simplicity reasons this application will only handle authentication and will use a memory-mapped list of users.

To get started, we will need to set up our project root directory. We need to create the directory pusher-channels-queue somewhere in $GOPATH. Ideally, this should resolve to $GOPATH/src/github.com/pusher-tutorials/pusher-channels-queue.

After doing the above, we will need to create a go directory since that is where our Golang application will live.

    $ mkdir go

The only external library we will need here are the Channel’s Golang SDK and a library to help us load our Pusher Channels keys. You can fetch that by running the command below:

bash
1$ go get github.com/pusher/pusher-http-go
2    $ go get github.com/joho/godotenv
Copy

To get started, you will need to create an .env file with the following contents:

bash
1// github.com/pusher-tutorials/pusher-channels-queue/go/.env
2    
3    PUSHER_APP_ID="YOUR_APP_ID"
4    PUSHER_APP_KEY="YOUR_APP_KEY"
5    PUSHER_APP_SECRET="YOUR_APP_SECRET"
6    PUSHER_APP_CLUSTER="YOUR_APP_CLUSTER"
7    PUSHER_APP_SECURE="1"
Copy

Once this has been done, we will need to create a main.go file.

go
1// github.com/pusher-tutorials/pusher-channels-queue/go/main.go
2    package main
3    
4    func main() {
5    
6            port := flag.Int("http.port", 1400, "Port to run HTTP service on")
7    
8            flag.Parse()
9            
10            err := godotenv.Load()
11            if err != nil {
12                    log.Fatal("Error loading .env file")
13            }
14    
15            appID := os.Getenv("PUSHER_APP_ID")
16            appKey := os.Getenv("PUSHER_APP_KEY")
17            appSecret := os.Getenv("PUSHER_APP_SECRET")
18            appCluster := os.Getenv("PUSHER_APP_CLUSTER")
19            appIsSecure := os.Getenv("PUSHER_APP_SECURE")
20    
21            var isSecure bool
22            if appIsSecure == "1" {
23                    isSecure = true
24            }
25    
26            client := &pusher.Client{
27                    AppId:   appID,
28                    Key:     appKey,
29                    Secret:  appSecret,
30                    Cluster: appCluster,
31                    Secure:  isSecure,
32            }
33            
34            mux := http.NewServeMux()
35            
36            mux.Handle("/login", http.HandlerFunc(login(client)))
37            
38            log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), mux))
39    }
Copy

In the above, we created an HTTP server that responds to the login route. We will go on to implement the login function subsequently.

Since we will be using a memory mapped list of users to prevent complications that might drive us away from the main focus of the tutorial. We will need to go ahead to create those. Paste the following code in the main.go file.

go
1// github.com/pusher-tutorials/pusher-channels-queue/go/main.go
2    
3    type User struct {
4            Email    string
5            Password string
6    }
7    
8    var (
9            validUsers = map[string]User{
10                    "admin": User{
11                            Email:    "youremail@gmail.com",
12                            Password: "admin",
13                    },
14                    "lanre": User{
15                            Email:    "youremail@gmail.com",
16                            Password: "lanre",
17                    },
18            }
19    )
Copy

You should replace youremail@gmail.com with your real email address so as to get the email when we get to the end of the tutorial.

Now back to the login function, you can go ahead to paste the following code in main.go

go
1// github.com/pusher-tutorials/pusher-channels-queue/go/main.go
2    
3    func encode(w io.Writer, v interface{}) {
4            json.NewEncoder(w).Encode(v)
5    }
6    
7    func login(client *pusher.Client) http.HandlerFunc {
8            return func(w http.ResponseWriter, r *http.Request) {
9                    defer r.Body.Close()
10    
11                    var request struct {
12                            UserName string `json:"userName"`
13                            Password string `json:"password"`
14                    }
15    
16                    type response struct {
17                            Message string `json:"message"`
18                            Success bool   `json:"success"`
19                    }
20                    
21                    // Make sure to only respond to the "/login" route
22                    // due to limitations in the standard HTTP router
23                    if r.URL.Path != "/login" {
24                            w.WriteHeader(http.StatusNotFound)
25                            return
26                    }
27    
28                    // Only HTTP posts are accepted
29                    if r.Method != http.MethodPost {
30                            w.WriteHeader(http.StatusMethodNotAllowed)
31                            return
32                    }
33    
34                    if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
35                            w.WriteHeader(http.StatusBadRequest)
36                            encode(w, response{"Invalid request body", false})
37                            return
38                    }
39                    
40                    // Check if the user exists in our memory mapped list.
41                    user, ok := validUsers[request.UserName]
42                    if !ok {
43                            w.WriteHeader(http.StatusBadRequest)
44                            encode(w, response{"User not found", false})
45                            return
46                    }
47                    
48                    
49                    // Do the passwords match ?
50                    if user.Password != request.Password {
51                            w.WriteHeader(http.StatusBadRequest)
52                            encode(w, response{"Password does not match", false})
53                            return
54                    }
55    
56                    w.WriteHeader(http.StatusOK)
57                    encode(w, response{"Login successful", true})
58    
59                    host, _, err := net.SplitHostPort(r.RemoteAddr)
60                    if err != nil {
61                            fmt.Fprintf(w, "userip: %q is not IP:port", r.RemoteAddr)
62                            return
63                    }
64    
65                    var ip = host
66                    
67                    if host == "::1" {
68                            ip = "127.0.0.1"
69                    }
70                    
71                    client.Trigger("auth", "login", &struct {
72                            IP    string `json:"ip"`
73                            User  string `json:"user"`
74                            Email string `json:"email"`
75                    }{
76                            User:  request.UserName,
77                            IP:    ip,
78                            Email: user.Email,
79                    })
80            }
81    }
Copy

While it is pretty easy to grok through the code above due to the inline comments, I will still like to go through the last few lines. Especially from Line 59.

• We get the IP of the user from r.RemoteAddr.

Please note that if you end up running something that does this kind of IP fetching in production, this might not be the right approach if your Go application is behind a proxy.

• We also check to make sure we have a valid IP address by making use of the net.SplitHostPort utility function.
• Then we finally publish the data to the auth channel.

At this point, the entire main.go should look like the following:

go
1// github.com/pusher-tutorials/pusher-channels-queue/go/main.go
2    
3    package main
4    
5    import (
6            "encoding/json"
7            "flag"
8            "fmt"
9            "io"
10            "log"
11            "net"
12            "net/http"
13            "os"
14    
15            "github.com/joho/godotenv"
16            pusher "github.com/pusher/pusher-http-go"
17    )
18    
19    type User struct {
20            Email    string
21            Password string
22    }
23    
24    var (
25            validUsers = map[string]User{
26                    "admin": User{
27                            Email:    "youremail@gmail.com",
28                            Password: "admin",
29                    },
30                    "lanre": User{
31    
32                            Email:    "youremail@gmail.com",
33                            Password: "lanre",
34                    },
35            }
36    )
37    
38    func main() {
39    
40            port := flag.Int("http.port", 1400, "Port to run HTTP service on")
41    
42            flag.Parse()
43            
44            err := godotenv.Load()
45            if err != nil {
46                    log.Fatal("Error loading .env file")
47            }
48    
49            appID := os.Getenv("PUSHER_APP_ID")
50            appKey := os.Getenv("PUSHER_APP_KEY")
51            appSecret := os.Getenv("PUSHER_APP_SECRET")
52            appCluster := os.Getenv("PUSHER_APP_CLUSTER")
53            appIsSecure := os.Getenv("PUSHER_APP_SECURE")
54    
55            var isSecure bool
56            if appIsSecure == "1" {
57                    isSecure = true
58            }
59    
60            client := &pusher.Client{
61                    AppId:   appID,
62                    Key:     appKey,
63                    Secret:  appSecret,
64                    Cluster: appCluster,
65                    Secure:  isSecure,
66            }
67    
68            mux := http.NewServeMux()
69    
70            mux.Handle("/login", http.HandlerFunc(login(client)))
71    
72            log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), mux))
73    }
74    
75    func encode(w io.Writer, v interface{}) {
76            json.NewEncoder(w).Encode(v)
77    }
78    
79    func login(client *pusher.Client) http.HandlerFunc {
80            return func(w http.ResponseWriter, r *http.Request) {
81                    defer r.Body.Close()
82    
83                    var request struct {
84                            UserName string `json:"userName"`
85                            Password string `json:"password"`
86                    }
87    
88                    type response struct {
89                            Message string `json:"message"`
90                            Success bool   `json:"success"`
91                    }
92    
93                    if r.URL.Path != "/login" {
94                            w.WriteHeader(http.StatusNotFound)
95                            return
96                    }
97    
98                    if r.Method != http.MethodPost {
99                            w.WriteHeader(http.StatusMethodNotAllowed)
100                            return
101                    }
102    
103                    if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
104                            w.WriteHeader(http.StatusBadRequest)
105                            encode(w, response{"Invalid request body", false})
106                            return
107                    }
108    
109                    user, ok := validUsers[request.UserName]
110                    if !ok {
111                            w.WriteHeader(http.StatusBadRequest)
112                            encode(w, response{"User not found", false})
113                            return
114                    }
115    
116                    if user.Password != request.Password {
117                            w.WriteHeader(http.StatusBadRequest)
118                            encode(w, response{"Password does not match", false})
119                            return
120                    }
121    
122                    w.WriteHeader(http.StatusOK)
123                    encode(w, response{"Login successful", true})
124    
125                    host, _, err := net.SplitHostPort(r.RemoteAddr)
126                    if err != nil {
127                            fmt.Fprintf(w, "userip: %q is not IP:port", r.RemoteAddr)
128                            return
129                    }
130    
131                    var ip = host
132    
133                    if host == "::1" {
134                            ip = "127.0.0.1"
135                    }
136    
137                    client.Trigger("auth", "login", &struct {
138                            IP    string `json:"ip"`
139                            User  string `json:"user"`
140                            Email string `json:"email"`
141                    }{
142                            User:  request.UserName,
143                            IP:    ip,
144                            Email: user.Email,
145                    })
146            }
147    }
Copy

Run the Go program:

bash
1$ cd $GOPATH/src/github.com/pusher-tutorials/pusher-channels-queue/go
2    $ go run main.go
Copy

You can try to send requests to the service with cURL by:

    $ curl  -X POST localhost:1400/login -d '{"username" : "admin", "password"  :"admin"}'

This will produce a response such as:

bash
1{"message":"Login successful","success":true}
Copy

Building the Node.js email service

We have made progress by publishing the events to Pusher Channels. You can verify that the events are published by looking at the Debug Console of the dashboard.

To build our Node.js email service, we will need to go back to the root directory, pusher-channels-queue. After which we will create the node directory as it will house our Node.js application.

    $ mkdir node

We will need a couple libraries for the application;

• pusher-js - the NodeJS SDK for Pusher Channels.
• nodemailer - We need this to send emails.
• dotenv - We need this to load environment variables from a file.
• handlebars - We need to dynamically replace contents of the email before sending it. Things like username and IP address come to mind here.
• fs - We need to be able to read the content of the email template from the filesystem. You can have a look at the email template here.

To install the above, you will need to create a package.json file that contains the following:

json
1// github.com/pusher-tutorials/pusher-channels-queue/node/package.json
2    {
3      "dependencies": {
4        "dotenv": "^6.2.0",
5        "fs": "^0.0.1-security",
6        "handlebars": "^4.0.12",
7        "nodemailer": "^4.7.0",
8        "pusher-js": "^4.3.1"
9      }
10    }
Copy

You will need to run npm install to get install those dependencies.

Since we need to subscribe to Pusher Channels, we need to first include the required values in .env.

bash
1// github.com/pusher-tutorials/pusher-channels-queue/node/.env
2    PUSHER_APP_CLUSTER="YOUR_APP_CLUSTER"
3    PUSHER_APP_SECURE="1"
4    PUSHER_APP_KEY="YOUR_APP_KEY"
5    MAILER_EMAIL="you@gmail.com"
6    MAILER_PASSWORD="Password"
Copy

Then create an index.js file

javascript
1// github.com/pusher-tutorials/pusher-channels-queue/node/index.js
2    
3    require('dotenv').config();
4    const Pusher = require('pusher-js');
5    const nodemailer = require('nodemailer');
6    const handlebars = require('handlebars');
7    const fs = require('fs');
8    
9    const pusherSocket = new Pusher(process.env.PUSHER_APP_KEY, {
10      forceTLS: process.env.PUSHER_APP_SECURE === '1' ? true : false,
11      cluster: process.env.PUSHER_APP_CLUSTER,
12    });
13    
14    const transporter = nodemailer.createTransport({
15      service: 'gmail',
16      auth: {
17        user: process.env.MAILER_EMAIL,
18        pass: process.env.MAILER_PASSWORD,
19      },
20    });
21    
22    const channel = pusherSocket.subscribe('auth');
23    
24    channel.bind('login', data => {
25       
26      fs.readFile('./index.html', { encoding: 'utf-8' }, function(err, html) {
27        if (err) {
28          throw err;
29        }
30        
31        const template = handlebars.compile(html);
32        const replacements = {
33          username: data.user,
34          ip: data.ip,
35        };
36    
37        let mailOptions = {
38          from: '"Pusher Tutorial demo" <foo@example.com>',
39          to: data.email,
40          subject: 'New login into Pusher tutorials demo app',
41          html: template(replacements),
42        };
43        
44        transporter.sendMail(mailOptions, function(error, response) {
45          if (error) {
46            console.log(error);
47            callback(error);
48          }
49        });
50      });
51      
52      console.log(data);
53    });
Copy

In the above code, we read the contents of index.html and process it like a handlebars template with handlebars.compile(html). This is because we are dynamically replacing {{ username }} and {{ ip }}.

So far, we have not created the index.html . You will need to create the aforementioned file and paste the following contents:

html
1// github.com/pusher-tutorials/pusher-channels-queue/node/index.html
2    
3    <!doctype html>
4    <html>
5      <head>
6        <meta name="viewport" content="width=device-width" />
7        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
8        <title>Simple Transactional Email</title>
9        <style>
10          /* -------------------------------------
11              GLOBAL RESETS
12          ------------------------------------- */
13    
14          /*All the styling goes here*/
15    
16          img {
17            border: none;
18            -ms-interpolation-mode: bicubic;
19            max-width: 100%;
20          }
21    
22          body {
23            background-color: #f6f6f6;
24            font-family: sans-serif;
25            -webkit-font-smoothing: antialiased;
26            font-size: 14px;
27            line-height: 1.4;
28            margin: 0;
29            padding: 0;
30            -ms-text-size-adjust: 100%;
31            -webkit-text-size-adjust: 100%;
32          }
33    
34          table {
35            border-collapse: separate;
36            mso-table-lspace: 0pt;
37            mso-table-rspace: 0pt;
38            width: 100%; }
39            table td {
40              font-family: sans-serif;
41              font-size: 14px;
42              vertical-align: top;
43          }
44    
45          /* -------------------------------------
46              BODY & CONTAINER
47          ------------------------------------- */
48    
49          .body {
50            background-color: #f6f6f6;
51            width: 100%;
52          }
53    
54          /* Set a max-width, and make it display as block so it will automatically stretch to that width, but will also shrink down on a phone or something */
55          .container {
56            display: block;
57            Margin: 0 auto !important;
58            /* makes it centered */
59            max-width: 580px;
60            padding: 10px;
61            width: 580px;
62          }
63    
64          /* This should also be a block element, so that it will fill 100% of the .container */
65          .content {
66            box-sizing: border-box;
67            display: block;
68            Margin: 0 auto;
69            max-width: 580px;
70            padding: 10px;
71          }
72    
73          /* -------------------------------------
74              HEADER, FOOTER, MAIN
75          ------------------------------------- */
76          .main {
77            background: #ffffff;
78            border-radius: 3px;
79            width: 100%;
80          }
81    
82          .wrapper {
83            box-sizing: border-box;
84            padding: 20px;
85          }
86    
87          .content-block {
88            padding-bottom: 10px;
89            padding-top: 10px;
90          }
91    
92          .footer {
93            clear: both;
94            Margin-top: 10px;
95            text-align: center;
96            width: 100%;
97          }
98            .footer td,
99            .footer p,
100            .footer span,
101            .footer a {
102              color: #999999;
103              font-size: 12px;
104              text-align: center;
105          }
106    
107          /* -------------------------------------
108              TYPOGRAPHY
109          ------------------------------------- */
110          h1,
111          h2,
112          h3,
113          h4 {
114            color: #000000;
115            font-family: sans-serif;
116            font-weight: 400;
117            line-height: 1.4;
118            margin: 0;
119            margin-bottom: 30px;
120          }
121    
122          h1 {
123            font-size: 35px;
124            font-weight: 300;
125            text-align: center;
126            text-transform: capitalize;
127          }
128    
129          p,
130          ul,
131          ol {
132            font-family: sans-serif;
133            font-size: 14px;
134            font-weight: normal;
135            margin: 0;
136            margin-bottom: 15px;
137          }
138            p li,
139            ul li,
140            ol li {
141              list-style-position: inside;
142              margin-left: 5px;
143          }
144    
145          a {
146            color: #3498db;
147            text-decoration: underline;
148          }
149    
150          /* -------------------------------------
151              BUTTONS
152          ------------------------------------- */
153          .btn {
154            box-sizing: border-box;
155            width: 100%; }
156            .btn > tbody > tr > td {
157              padding-bottom: 15px; }
158            .btn table {
159              width: auto;
160          }
161            .btn table td {
162              background-color: #ffffff;
163              border-radius: 5px;
164              text-align: center;
165          }
166            .btn a {
167              background-color: #ffffff;
168              border: solid 1px #3498db;
169              border-radius: 5px;
170              box-sizing: border-box;
171              color: #3498db;
172              cursor: pointer;
173              display: inline-block;
174              font-size: 14px;
175              font-weight: bold;
176              margin: 0;
177              padding: 12px 25px;
178              text-decoration: none;
179              text-transform: capitalize;
180          }
181    
182          .btn-primary table td {
183            background-color: #3498db;
184          }
185    
186          .btn-primary a {
187            background-color: #3498db;
188            border-color: #3498db;
189            color: #ffffff;
190          }
191    
192          /* -------------------------------------
193              OTHER STYLES THAT MIGHT BE USEFUL
194          ------------------------------------- */
195          .last {
196            margin-bottom: 0;
197          }
198    
199          .first {
200            margin-top: 0;
201          }
202    
203          .align-center {
204            text-align: center;
205          }
206    
207          .align-right {
208            text-align: right;
209          }
210    
211          .align-left {
212            text-align: left;
213          }
214    
215          .clear {
216            clear: both;
217          }
218    
219          .mt0 {
220            margin-top: 0;
221          }
222    
223          .mb0 {
224            margin-bottom: 0;
225          }
226    
227          .preheader {
228            color: transparent;
229            display: none;
230            height: 0;
231            max-height: 0;
232            max-width: 0;
233            opacity: 0;
234            overflow: hidden;
235            mso-hide: all;
236            visibility: hidden;
237            width: 0;
238          }
239    
240          .powered-by a {
241            text-decoration: none;
242          }
243    
244          hr {
245            border: 0;
246            border-bottom: 1px solid #f6f6f6;
247            Margin: 20px 0;
248          }
249    
250          /* -------------------------------------
251              RESPONSIVE AND MOBILE FRIENDLY STYLES
252          ------------------------------------- */
253          @media only screen and (max-width: 620px) {
254            table[class=body] h1 {
255              font-size: 28px !important;
256              margin-bottom: 10px !important;
257            }
258            table[class=body] p,
259            table[class=body] ul,
260            table[class=body] ol,
261            table[class=body] td,
262            table[class=body] span,
263            table[class=body] a {
264              font-size: 16px !important;
265            }
266            table[class=body] .wrapper,
267            table[class=body] .article {
268              padding: 10px !important;
269            }
270            table[class=body] .content {
271              padding: 0 !important;
272            }
273            table[class=body] .container {
274              padding: 0 !important;
275              width: 100% !important;
276            }
277            table[class=body] .main {
278              border-left-width: 0 !important;
279              border-radius: 0 !important;
280              border-right-width: 0 !important;
281            }
282            table[class=body] .btn table {
283              width: 100% !important;
284            }
285            table[class=body] .btn a {
286              width: 100% !important;
287            }
288            table[class=body] .img-responsive {
289              height: auto !important;
290              max-width: 100% !important;
291              width: auto !important;
292            }
293          }
294    
295          /* -------------------------------------
296              PRESERVE THESE STYLES IN THE HEAD
297          ------------------------------------- */
298          @media all {
299            .ExternalClass {
300              width: 100%;
301            }
302            .ExternalClass,
303            .ExternalClass p,
304            .ExternalClass span,
305            .ExternalClass font,
306            .ExternalClass td,
307            .ExternalClass div {
308              line-height: 100%;
309            }
310            .apple-link a {
311              color: inherit !important;
312              font-family: inherit !important;
313              font-size: inherit !important;
314              font-weight: inherit !important;
315              line-height: inherit !important;
316              text-decoration: none !important;
317            }
318            .btn-primary table td:hover {
319              background-color: #34495e !important;
320            }
321            .btn-primary a:hover {
322              background-color: #34495e !important;
323              border-color: #34495e !important;
324            }
325          }
326    
327        </style>
328      </head>
329      <body class="">
330        <table role="presentation" border="0" cellpadding="0" cellspacing="0" class="body">
331          <tr>
332            <td>&nbsp;</td>
333            <td class="container">
334              <div class="content">
335    
336                <!-- START CENTERED WHITE CONTAINER -->
337                <table role="presentation" class="main">
338    
339                  <!-- START MAIN CONTENT AREA -->
340                  <tr>
341                    <td class="wrapper">
342                      <table role="presentation" border="0" cellpadding="0" cellspacing="0">
343                        <tr>
344                          <td>
345                            <p>Hi {{ username }},</p>
346                            <p>You’ve successfully signed into the demo app.</p>
347                            <p>You signed in from the IP address, {{ ip }}</p>
348                            <table role="presentation" border="0" cellpadding="0" cellspacing="0" class="btn btn-primary">
349                              <tbody>
350                                <tr>
351                                  <td align="left">
352                                    <table role="presentation" border="0" cellpadding="0" cellspacing="0">
353                                      <tbody>
354                                        <tr>
355                                          <td> <a href="https://pusher.com"
356                                                          target="_blank">Visit
357                                                          Pusher</a> </td>
358                                        </tr>
359                                      </tbody>
360                                    </table>
361                                  </td>
362                                </tr>
363                              </tbody>
364                            </table>
365                          </td>
366                        </tr>
367                      </table>
368                    </td>
369                  </tr>
370    
371                <!-- END MAIN CONTENT AREA -->
372                </table>
373    
374    
375              <!-- END CENTERED WHITE CONTAINER -->
376              </div>
377            </td>
378            <td>&nbsp;</td>
379          </tr>
380        </table>
381      </body>
382    </html>
Copy

We listen for the login event and pick out the important data from there. In this case, the user’s name and IP address from which they logged in. After which we send the email to the user.

You will need to start the Node.js service by running node index.js. After doing that, you can send login requests to the Golang service again.

You should check your email:

NOTE: You might need to allow insecure apps

Conclusion

In this tutorial, we have leveraged Pusher Channels as a messaging queue between two different applications. While we used this to send email notifications, we can use this for much more interesting patterns depending on your application’s needs.

The entire source code of this tutorial can be found on GitHub.