Approaching Realtime as a Blank Slate

When Pusher started as a company the main environment in which WebSockets were used was the web browser. Since the birth of WebSockets many new JavaScript runtimes have emerged. Now JavaScript runs everywhere, including servers and mobile applications which could benefit from a WebSocket connection. Unfortunately, those environments differ in several ways, which make web browser libraries incompatible.

Because our main JavaScript library – pusher-js – always targeted only web browsers, we forked it and undertook an experiment with this new library to make it compatible with other environments, such as Node.js or React Native.

So let’s start at the beginning.

Identifying environmental differences

Node.js and modern web browsers provide very similar JavaScript runtimes, but the application environments differ in a number of key ways.

All modern web browsers expose an XMLHttpRequest API for sending HTTP requests and most of them support the WebSocket API. On the other hand, Node.js provides a rich standard library that includes TCP and HTTP clients, but it doesn’t include a WebSocket client and the HTTP library has an interface incompatible with XMLHttpRequest.

There are also plenty of other non-critical web browser APIs used in client libraries, the most notable being the Document Object Model, used to implement JSONP and iframe-based WebSocket polyfills. While useful for improving compatiblity for old browsers, they are very difficult to replicate in non-browser environments.

Removing the pain points

The first major decision we made was to drop support for very old web browsers, the worst offenders being old versions of Internet Explorer. Those browsers constitute a minority of the market, but incur the majority of complexity in pusher-js. Additionally, modern, rich web applications are usually incompatible with those browsers.

We decided that the baseline would be a browser supporting cross-origin HTTP requests via XMLHttpRequest (including XDomainRequest in Internet Explorer 8 & 9). Thanks to this, we could remove several parts of the library:

• SockJS fallbacks (we use our own HTTP fallbacks)
• JSONP authentication (since we assume cross-origin XHR is available)
• JSONP stats reporting
• dependency loader (for fallback transports and JSON polyfill)

After removing those components, the library became much simpler, HTTP fallbacks faster and we didn’t have to worry about DOM and other very-browser-specific APIs.

Abstracting other differences

Even after dropping support for old browsers, there are still some web browser APIs that need to be used in the library. To make it harder, those interfaces are not supported in some browsers (e.g. WebSockets in IE8). In order to make our code compatible with other environments, we needed to find replacements for those APIs.

First, we identified 3 main APIs that need different approaches in particular JavaScript environments:

• WebSocket API
• XMLHttpRequest
• NavigatorOnLine API

Then we created three modules, each with different implementation of above APIs:

• pusher-websocket-iso-externals-node
  • uses faye-websocket and xmlhttprequest npm modules
  • has no online detection
• pusher-websocket-iso-externals-web
  • uses browser’s native WebSocket and XMLHttpRequest classes
  • uses window.navigator for online detection
• pusher-websocket-iso-externals-react-native
  • similar to the web module, but with disabled online detection (to be implemented using React’s own API)

For example, the WebSocket module file for Node.js contains following code:

javascript
1module.exports = require("faye-websocket").Client;
Copy

The library uses the Node.js module by default. This way, we don’t need a build step for Node.js application.

Building the library for the web

So far we’ve made the library compatible with Node.js, but we can’t use it in a web browser, because it requires NPM modules for WebSocket and XMLHttpRequest clients.

But because those APIs are now wrapped in modules, we can simply replace those Node.js modules with ones for the web.

For example, here’s the source of the WebSocket module for the web environment:

javascript
1module.exports = window.WebSocket || window.MozWebSocket;
Copy

Now we need to bundle all that code into one JavaScript file to be distributed to web browsers. We decided to use webpack for that purpose.

Webpack is a JavaScript bundler, which supports CommonJS modules, so it can be used for Node.js code. One of its features is the ability to replace whole modules with arbitrary code or different modules, which is very handy in our case.

Let’s jump straight to the webpack configuration file for the web environment:

javascript
1var path = require("path");
2var NormalModuleReplacementPlugin = require('webpack').NormalModuleReplacementPlugin;
3var version = require('../package').version;
4
5module.exports = {
6  entry: "./src/pusher",
7  output: {
8    library: "Pusher",
9    path: path.join(__dirname, "../bundle/web"),
10    filename: "pusher.js"
11  },
12  externals: {
13    '../package': '{version: "'+ version +'"}'
14  },
15  plugins: [
16    new NormalModuleReplacementPlugin(
17      /^pusher-websocket-js-iso-externals-node\/ws$/,
18      "pusher-websocket-js-iso-externals-web/ws"
19    ),
20    new NormalModuleReplacementPlugin(
21      /^pusher-websocket-js-iso-externals-node\/xhr$/,
22      "pusher-websocket-js-iso-externals-web/xhr"
23    ),
24    new NormalModuleReplacementPlugin(
25      /^pusher-websocket-js-iso-externals-node\/net_info$/,
26      "pusher-websocket-js-iso-externals-web/net_info"
27    )
28  ]
29}
Copy

You can see that we use NormalModuleReplacementPlugin to replace Node.js modules with their web browser counterparts. Now we just need to run webpack with above configuration:

javascript
1node_modules/webpack/bin/webpack.js --config=webpack/config.web.js
Copy

and now we can use the pusher.js file from dist/web directory to connect to Pusher from web browsers.

How it works in reality

You can check out the code from GitHub: https://github.com/pusher-community/pusher-websocket-js-iso. Please keep in mind that the library is experimental – it hasn’t yet been tested thoroughly enough to be used for production applications.

We have also published the library to NPM, so you can simply install it using following command:

bash
1npm install pusher-websocket-iso
Copy

For Node.js applications, require the module to get the Pusher constructor:

javascript
1var Pusher = require("pusher-websocket-iso");
Copy

For web browser usage, you can build the files manually, but we also bundle them in the NPM module. There are two bundles in the module directory:

• dist/web/pusher.js – exports the Pusher constructor to the global scope
• dist/web-umd/pusher.js – exports a UMD-compatible module

Special thanks to one of Pusher’s developers Jamie Patel for his work on this. We’d love to hear how you’re getting on and whether anyone else has been experimenting in this way.