Sessions is temporarily moving to YouTube, check out all our new videos here.

Fullstack Integration Testing That Doesn't Suck

Artem Avetisyan speaking at London Node User Group in March, 2017
Great talks, fired to your inbox 👌
No junk, no spam, just great talks. Unsubscribe any time.

About this talk

With the help of Electron, it is now possible to run your entire web stack (SPA + backend) in a single process. When applied to testing, this becomes a compelling alternative to webdriver based testing, as it is much faster and easier to debug. In this talk, we will look at what types of tests there are in a typical web app, what they're for and then show off a real example.


Tonight I'm going to be talking about something unheard of, something that actually very few people believe is even possible. And that is full stack integration testing that doesn't suck. At Featurist we take testing very seriously, which means we write all the tests. And we also take a lot of care about our development experience. And if something doesn't feel right or is painful, we experiment with the tools, try this and that, and there's nothing good out there, we will end up writing our own stuff. And I think when it comes to integration testing of web applications, we have arrived to a particularly sweet spot, and that's what this talk is about. But before we get into the details, let's quickly establish some base terminology in terms of what types of testing are there. Broadly speaking, there are unit tests and integration tests. Unit tests are focused on taking a unit of code such as modular class, and run a test in isolation, testing its behavior in isolation, providing inputs and asserting outputs. Integration test in contrasts are treating your whole system as a whole, running tests across the entire stack. Those tests have their pros and cons. The good things about unit tests, they're easy to write. Since the code on the test is very easy to set up, normally it's just instantiating a class or importing a modular function, they're very fast, they run in single process and incredibly fast to excise, and have a good feedback. By which I mean, if anything goes wrong it's typically easy to understand what has gone wrong, since there's just not much going on in a single test. The not so good things about unit test is that get small coverage. So if you have a sufficiently large system, a single task doesn't cover much, it's not very interesting normally. Unit tests also don't cover the glue code which is the code that is used to put the units together. And again, in a system of any size there's a lot of that code. And finally, the user interface is typically hard to unit test because where the UI code runs in the context of this massive global API called DOM, and unless you're using a particular framework such as React, that sort of remedy that problem, it's kind of hard to have a unit. So there're integration tests to compliment those bad things. So the good things about integration tests, they cover is lots of code. A single test casts through the entire system, all the layers, or lots of the layers, and that's why it's a valuable thing. It also covers glue code by definition since it exercise everything, glue code included. And it covers the UI, since, well that's how users exercise our systems normally, therefore that's how integration test exercises system as well. And the not so good things about integration tests is that they're very complex to set up. Systems normally include databases and some session state, and what have you, and to recreate a particular test in a particular place of an application sometimes it can be just really hard. They're slow, they're slow to start and they're slow to run. They're slow to start because again applications consist of many layers, be it a browser application, and then there's a backend application, there's a database, and you have to bring it all up and run it synchronously-ish, and it's hard to debug. Because there's so many layers underneath just the browser window and browser is everything the test has to, mostly, that's how...that's where the feedback is. For example, if somebody is booking a ticket, and something went wrong, there is no confirmation, ticket booked successfully, and then you have to go and figure what exactly happened. But all you have to start with is just lack of a div in a browser. So yeah, we have unit test, they're nice and easy, they're the joy to work with, but sometimes not very useful, it's called good cop. And then we have integration test, they're tedious, slow, and painful. But try leaving those aside, those integration tests aside, and it's only a matter of time till you get into this. The slide is called, all unit tests are passing, no integration tests. Beautiful. So yeah, on a typical project therefore we have few integration tests, because while you can get away without having anything at all, but they're too painful to write lots of them. We have lots of unit tests to cover all the permutations that's underneath, and we have mixtures of the two in between, be it testing of some adapters, testing of queries, or may be functional tests of particular components. And, years ago smart people already noticed that and came up with this notion of test pyramid, where we have a UI, where we have expensive and slow tests at the top and then some mixture of the slow and fast test in between, and finally fat base of unit tests. But what if, what if I told you that in JavaScript today this isn't quite true anymore. The test could be a lot less painful, and they could also be faster. So let's have a look again at the bad things of integration tests. We have a complex setup, there's nothing much we can do that here. We have slow and hard to debug. Those last two things are not inherent part of integration tests as such, they are at least partly implied by the tools we're using to write those tests. And the tools that are currently mainstream to write those tests in JavaScript and in many other web technologies is Webdriver IO which in JavaScript is a Selenium. A Selenium binding is for NodeJS. And if that doesn't tell you much, it's okay, I still don't understand what that means too. But, I know what it does, and what it does is a very simple thing, it allows you to control the behavior of a web page such as clicking buttons or watching things being on the page and not being on the page from a remote process, for example from a script. And that very neatly applies to the problem of integration testing since we want an automated way of exercising the UI. So a typical Webdriver setup is as running your backend application in one process, it's running your client side JavaScript in another process, in a browser process, and it's also running tests in a third process. So in a test we have three of those processes all running together, coordinating the run. And that's where the slowness and the hard to debug bit come from. Because for example, if you we have a client's application that is using common JS requires, then before it gets to the browser process of course we need to run through for example browserify, the pre which is slow. And then if you want to debug or if you want to pause the entire test, then you need like three breakpoints, you need to coordinate the breakpoints to pause the test in a particular place, is just painful. And that brings us to the solution of this problem, and the solutions called Electron. And in a nutshell, Electron is a technology that allows us to let...that gives an access to a DOM API and to the node API in the same process. And what that means is that, on line one you can create a div, and on line two you can create a file of the same script. Or, on line one you can mount your React application, and then line two you can start your Express backend, and on line three you can seed your database. And that's exactly what I'm about to show you. Okay, so the test, we have a very simple app here. This app has a button, and when we click the button it calls express with service and fetches to dos. Here's the execus[SP] call that does that. And our test is going to do exactly that. It's going to click the button and assert that the data from the server, from the database actually is showing on the page. Let's run it in...okay, that's actually already finished although you didn't see it. It was really fast, but we can run it again. And this is less than a second, and the test is doing exactly what I said. In most client's side application it starts up the Express server, and seeds the database as well, all within the same process and all very fast. Let's look at the test here and put some breakpoints and see how we step through the code. Let's run the test again. And here is you can see we're about to click the button, and let's go and open up the browser application. And when he clicks the button, and here where you have to trust me, it's going to end up here where it makes this request to the Express API to fetch the data. So let's carry on, here we are, this is the browser application. And now let me show you something cool. Let me go into the actual server application which is the Express endpoint, which is right where we query the database, the SQL database on the server, and we return the data, here we are. Once again, this is the full stack, as full as you can get. And our entire test is paused, waiting patiently for us to examine for example the data we've just returned from the database. And then we can go here, and right before we show the data on the page, you can go back to the old browser app. Here we are, we're about to show the same data, this is a browser app already. And the test is passed. So this is possible to do with Electron. And this test is written in Mocha, and as electron-mocha, the library allows us to run those tests in Electron. But also a another crucial bit to the whole setup, is a way to interact with the DOM, by which I mean the way to assert that certain things that are in the DOM, and the way to click the button for example. And this is another...this is the library that we developed at Featurist, and we're using quite extensively for quite a long time already called Browser-Monkey. And it allows us to do exactly that, allows you to assert the DOM and interact with the DOM. It looks like this here. Which looks a little bit like JQuery if you really think about it, with one really important and critical difference, Browser-Monkey has implicit weights embedded into it. For example, right after we click the button here on line 45, when we arrive in line 46 there is no data still on the web page. So if you were using JQuery for example, this assertion would fail, there wouldn't be no one and two on the page. But Browse-Monkey knows that things in single page applications don't always are there yet, so it's going to wait implicitly for stuff to appear. The reason it's fast is that there is no need to browserify or run your client side code through the webpack or something. Since Electron has the DOM API as well as the...sorry, has the Node API as well as DOM, we can just use our require statements and stuff just works. So it's really, really, really fast compared to Weddriver based tests. In fact, I'm going to give you some numbers as to how fast it is. So, it's always fast the second time. But under three seconds and here we have two full stack test that seed the database, start the Express services, mount the app, and do everything including clean up and setting up. And that's pretty awesome I think. And it doesn't stop there, for example we can do something like running our toss test in vdom, it's not really Electron related, but it's just so cool so I can't just skip that. Look, it's one second, that's almost approaching those B2D threshold of 300 millisecond being a perfect feedback time. Okay, not quite, almost, but it's still good. Oh, okay, this is the demo. Right, so yeah. The test stack is largely speaking electron-mocha and browser-monkey. So those two things put together allow us to write these otherwise historically painful tests in a much, much more comfortable way. And it's not just the test, it's not just a test project in the side, we are using this for real on real client site projects. We are running tests in the Electron using electron-mocha, and browser-monkey too. So there's nothing that should stop anyone here to just go and do it tomorrow. And, well, finally, final note is that the fact that integration tests suddenly are easy to write doesn't mean that we should abandon all other types of test, [inaudible 00:18:07] still a valuable thing for particular cases. For example, where you need to exercise permutations of a particular behavior or logic. And, well this is it for me. Thank you. Yeah? - [Male 1] So, on the client project where you said running 1,000 or whatever tests, how much time does that take? How much time does that suite takes? - Well, we have a split there. We have s test that run only browser test, moching out the APIs, and this is a very, very large suite of tests, about 1,000 tests, and it runs in electron-mocha, it takes about maybe four minutes. Well, obviously it very much depends on what types of test they are, they're quite complex tests, not perfect. And we have proper full stack tests that run about two or three minutes, they're lesser then there but maybe 100, but there's more that they... They're not limited to just one Express app, they're like at least five or six of them, including some moched services. So yeah, that's few minutes which is quite good. But again you'd have to go and see the particulars of the test to make some meaning of those numbers. - [Male 2] But 1,000 test is pretty large. - Yeah. - [Male 3] [Inaudible 00:19:49]. Have you done any time or space complexity analysis on this? So how does it scale relative to doing integration test in other ways to try and test the [inaudible 00:20:03]. - No, we haven't. - [Male 4] Can you just repeat the question. - Oh, sorry, yes. Repeat the questions. The question was, have we done any.... - Complexity analysis... - Complexity analysis. - I'm just wondering how it scales relative to everything else, because we'll obviously having two tests running the same, it pretty impressive, but you've got the whole stack running on that. But when you have, let's say when you have to scale it up to real world test suites, does it scale comparatively to existing tools and methods that we use already? Or is it like, nice, but might vary? - Well, those tests, the way we're running tests, we're running them sequentially, so one after another. So I suppose the way we're using it it doesn't really matter if there's 1 or 1,000, it's just going to run one-by-one. And every test tears down completely everything and brings everything back up... - Yeah. - ...mostly. - So we haven't spotted any scalability issues so far. - Yeah, I was just interested to see a graph go from one test - this is how long it took - up to maybe 10,000 tests of a similar suite across the [inaudible 00:21:20] integration test tools. - Okay, well that's probably for the next version of this talk. - No, [inaudible 00:21:27] nice one, cheers man. - [Male 5] It's kind of at the moment, if you have basically in the web driver, [inaudible 00:21:37] do you have implicit ways, you have actually everything executed on load, and then you can add extra weight, what's the purpose of this writing tool? [Inaudible 00:21:50], but what is the purpose really? - The purpose of... - Because you have everything built in a web driver tools at the moment. So the benefit you said basically get there... - The benefit is... - [Inaudible 00:22:07]. - The benefit is the speed. The way web driver works fundamentally is different from the way you can run those tests in Electron. Web driver has it at least three separate processes, and which makes it slower and also harder to debug. For example, if you need to pause a test the way I showed like in your Express app, you'd need to coordinate perhaps breakpoints or you'd need to dance a little bit. But here you can just put a breakpoint and everything shuts there. And yeah, and like I said since there's no browserify role or webpack, there's no browser compilation at all., the startup time of those tests which is important, if you carry on repeating tests time after time while you're working on a feature, it's really important that it starts up quickly. And I think it's a very impressive gain there, so those are the things. When I talking about browser-monkey and implicit weights and stuff, yes a way it mirrors what web driver is giving you as a similar API for asserting and manipulating or interacting with the page. But since we're not using web driver because of its disadvantages that we don't really like, we had to come up with a similar tool that allows us to interact with the page, and [inaudible 00:23:46]. - [Inaudible 00:23:47]. - Okay. - [Male 6] Can you manage multiple browsers? - This is a very good question. This is one area where this approach falls short, of course you can't manage the multiple browsers, or at least I don't know how to do that, maybe you can. But I suppose the argument here would be that, largely we are interested in exercising behavior, and therefore if we want to test the browser...the code in different browsers, we'd just end up with a different suite of particular tests, a test that slice the [inaudible 00:24:28] and nothing else. And so that would be my excuse. Thank you very much.