Sessions is temporarily moving to YouTube, check out all our new videos here.

Testing against Time - Meaningful testing in Ember apps when timing matters

Jessica Jordan speaking at EmberFest in October, 2017
Great talks, fired to your inbox 👌
No junk, no spam, just great talks. Unsubscribe any time.

About this talk

Ember offers a rich API and a wide set of helpers and blueprints to make testing in your apps fast and straight-forward. But as the applications we write tests against grow more complex, we might find ourselves stumbling into test timeouts, eventually succeeding tests and other hard-to-reason about test errors brought on by asynchronous and time-dependent behaviour. This talk will give an insight into how to create meaningful test cases for async or other, time-related operations in our applications. We will see how we can leverage Ember's test helpers, newest JavaScript features and Ember community addons to reduce non-determinism in our test suites and make those tests turn green even as time passes by.


Hi everyone, my name's Jessica, and today I want to talk with you about testing in Ember apps, and specifically, how we can actually manage cases in which time is relevant in our application, how we can accommodate this in our tests, and also how we can write better and more meaningful tests in our application in regards to timing. And before we get started with the topic, I just want to quickly give you a short introduction about myself. I'm currently working as a software engineer at simplabs, a consultancy based in Munich. You might have already heard about yesterday. I'm also an editor at the Ember.js Times, so the big news that you might already be getting are coming from us and the Ember.js learning team which I also just recently joined, and if you haven't subscribed to it yet, I can highly recommend that you do, because it's like a really cool update about all the cool new things that happen in Ember currently, and you get fresh updates every week. So check it out if you haven't already. And last but not least, I'm also one of the co-organizers of the Ember.js Berlin Meetup, together with Clemens and also Joschka, who's organising also here at EmberFest. We are actually running this meetup every month, so if you at any time are in the city around, or if you are from Berlin, please feel free to join us, and also if you have some ideas for talks and would like to present something, we also very much appreciate any kind of like new speakers at our meetup. And last but not least, I also want to announce we have really cool stickers already, so if you want a really cool fancy hipster hamster, please, I can refer you to Joschka who has the bag with them. So with that said, let's actually get into the topic of this talk. If we think about testing, we might first of all wonder to ourselves sometimes, spending lots of time and work actually writing tests, why we actually do this, what our main motivations for actually writing reasonable tests, and spending time making them reliable and meaningful. And what we might find is probably the most important reasons that we first of all want to actually work with cool and fancy new testing frameworks, right. This is like really important. Also it's pretty to kind of like push back release dates and get like a nice communication style with other teams in our company. And last but not least, we actually have a great opportunity to spend even more time at work, right? So these are all really great reasons. But if we think about it a little bit further, we realise maybe the actual reason might be that we actually want to have confidence that anytime we build a new feature, or we ship something to our production website, that this experience that we created is actually reliable for our users and will just work, and we don't have to wonder about a call on the weekend that we have to fix a bug about something we just recent shipped, we just are confident that it just works. Also we have a great opportunity to actually secure our code base by the introduction of new bugs down the road. We can actually make sure with a suitable test suite that we test our code base against regressions in any kind of like manner, and therefore make life easier not only for other developers in our team, but also to our future selves down the road. And last but not least, it's also a great way to actually document our code base. We can actually give us and other people working on the code base a good idea what the intent is of a feature that we have just implemented, and also if you later look back on it, we have still a very good understanding what we did there when we implemented it for the first time. So that's a lot of great reasons to actually do testing. Then we actually have our Ember app, and we actually have really great tooling to do testing right off the works every time, a great component, or a new route to actually get like out of the box with EmberCLI, a new test file already for me ready to write a test. And so I'm really confident actually and I'll dive into the adventure and then it can be that things happen, right, and then if I write my first test, it might be that I actually get like failing tests, and it might be that oftentimes I just have like false negatives, so like assertions of which I know they should actually pass and that should actually be okay, because I manually tested a feature in my application, but it just doesn't seem to work on the tests, so I want to know okay, what's happened here? Also I might run into test timeouts. So any kind of like time I introduce a new feature, and I want to test it, I might either locally or maybe in like in my CI integration realise, oh, the tests are just like timing out, something seems to be wrong, but I would have to further debug that. And also, quite annoyingly, sometimes I have a lot of flakey tests, tests that sometimes pass on my machine, but then my coworker might check it out and go like, oh this actually like a failing test, right, and I have to check it again. And this also kind of like crashes a little bit my confidence in the tests that I wrote, because flakey test is maybe even worse than just downright failing tests, because I might just realise that it's sometimes failing much later. All of these things I actually, if they happen in my test suite, a good sign that things are in a very unacceptable condition and what I actually want to do is have more confidence in the test that I write and the code base, and also make it more reliable and meaningful. And with that, I now have to dig further and actually find out what are the problems, what are the reasons that my tests fail. And what I've found in my past experience is that oftentimes this has something to do with timing in my application. So many times it's about asynchronous operations, it's about setting times and also any kind of like things that I might not directly see being set in my application but that actually happen down the road when either my app runs or my tests run. And this makes it interesting for me to talk about that and to get like to dive into that let's maybe have a look at asynchronous operations first. If we are working with asynchronous operations, we oftentimes just want to be able to actually let our tests wait for something to happen, right? So let's maybe have a look at actually like a proper example to get into this topic. So for the newsletter, we are writing like, I create like a small demo app, and it's just like an overview page of all the things that happened recently in the last week, like around like Ember, community ripples and it's actually kind of like nice now not to have to browse through all the different repositories and find out okay, what is happening right now, right. So just like very simple app with just like this really nice rendering of pull requests and descriptions. And what we could now look into is actually how could we actually test something asynchronous, we already are familiar with, how could we test a route transition. If we would like to do that, we already know that our routes provide a model hook, and this will usually return a promise, and also in this instance, I actually would like to fetch all the different repos, the kind of like data models around the repos from the different organisations, so like Ember.js or EmberCLI and all the others, and then at the end I would actually like try to wait for all of these promises to resolve with the All invocation, and All is like part of the RSVP package and will actually like then resolve when these promises, the several ones have resolved. So this sounds pretty straightforward, actually, right? So let's actually test this in an acceptance test for example. What we could do is then use the visit helper. The visit helper actually helps us to exactly, specifically test the transition between different routes, and if we have a look into what this visit helper actually does inside, we realise it does something like looking up the router. It will also help us to boot up the app, doing some like setup work that we need to actually get this test running, and also it will help us to actually set up the router so it's available for transitions during our test. And last but not least, this will at the end return the wait helper. The wait helper actually is where the magic comes in of many other asynchronous helpers as we will see in this talk, that it will actually help us to execute all of this and let our tests wait at the specific time point until we actually go onto the next assertion or the next interaction that we want to do in our test. So with that, we might already be familiar with that, we can simply write acceptance tests to just test the transition into the route by using the visit helper, visiting the overview route, and then later on with an andThen helper actually wrap our assertions so we can make sure this actually runs through. Okay, this is pretty straightforward, right, so we might go like, okay, I kind of got this, okay, cool. Then we can have a look also at the documentation, we realise okay, there are a lot of other cool asynchronous helpers I can use in acceptance tests, there's visit, there's click, there's fillIn, keyEvent, many that I can use to actually interact with my application. So that's pretty nice. And the cool thing about all of them is they also do something very similar to the visit helper, they will actually just return a call to the wait helper, and therefore make sure that everything in my test suite runs smoothly as long as I use these helpers. So that's quite nice. So now I heard somewhere I shouldn't write acceptance tests only, right? So I try to write something about my components, a nice integration test. And for this example, let's have maybe a look at the actual demo app again. So if we for example wanted to test in my component, a nice functionality, you can click on a speech bubble and then it will actually load from the GitHub API, the different comments that are associated with the pull request right, because sometimes I want to write some nice more detailed comment into the newsletter, then it will look something like this, it will log the comments, and then I also would like to actually set these comments that I get back from the API into my component. So I do this in this instance. And also, I use the bounce to actually make sure that if people repeatedly hit that, I don't have like repeated requests to the API. Also, what we would like to do is actually, yeah, here's like actually the loading comments function as we can see here, and where we actually then later on set the comments as I already explained, and then we can actually get started, we can actually write our test. So now we would do something like this, right. We would go like, okay, we set up our test maybe with a generator for the component test. We might also use an add-on for example like ember-data-factory-guy, you already heard about it today, to actually like mock data that we can use in our component, because it has to receive the pull attribute, and also later on we actually want to click on the speech bubble, which we can now see in this example, is marked with the data test load comments attribute, and in this instance, for example, I'm using an add-on to actually add data test selectors to my component and make it available in my test. You can also check it out in the simplabs repo. And last but not least, also I will actually do the assertion and check, okay, did it actually load, did it actually get like different numbers to actually display my UI, right? So this seems pretty straightforward. But then again there was something right. This actually fails because, oh yeah right, if I actually do the request to the API, then it might take some time, and at this point it might be that when the assertion actually is executed, the promise hasn't resolved yet at all, right. So this is kind of like difficult. Also, another issue I might stumble into is that now I actually do like a real request from the API, right. Should probably also not happen in my test case, right, it should actually be self-contained. So to avoid all of this, let's maybe check how we can mock that as well. How can we actually mock this request in our test. And for that I would like to present one opportunity to do that, there are probably other libraries out there that you can use, but I really like this one, and it's called testdouble, and with a simple yarn add testdouble, you can just insert it into your application and make use of that, and actually mock things like these API calls. So if we for example now, add this we can do something like this, we can actually create, yeah this is also important for us to notice we can actually register a service in our test because in our component we might also have like a request server that actually does like all the fetching for us, and this makes it now in our tests actually nicer to stub it, right, just so that you know. And in this service we can actually assign a td function to our fetch in the service, so this is what we usually would use in our component to actually do the call. And this is really nice because now that we actually stub that, we can make use of it later on in our test when we actually want to mock that and actually wait for okay, did our component when we actually clicked that button really do the request? And with the td ren API that you can see here in this test case, we have the opportunity now to say, okay, anytime when this function is called in this way, also with this arguments, so like with comments URL, then please resolve with the comments that we already have marked, right. And with this we can actually like directly resolve the promise that should be initiated through this function and make sure that actually the data is available once the actual promise in our test resolves. This is really nice. So we set all of this up, we render our component, and then later on we actually click again on the button, and then we do the assertion. So this should actually kinda work, right. You can also see like here again, in comparison how it actually works, so we would have the fetch call, right, and this is mapped to the actual mocked call in testdouble. And just to, like the first explanation so we get a little bit more background, you also have like other APIs available, so you don't only have one like forward then resolve, but also for then reject, and also if you want to mock any kind of callbacks, you can easily do this with testdouble as well. This is really nice if you have like different other use cases you want to cover. But now that we've done that, we realise okay, something's still kind of like missing, it'd kind of like still not really working, and what we might realise if we have a look at this again is that if we actually click, then there's actually a promise returned from our component, and this still has to resolve. So if if we do the assertion and it's still too early, right. So we would like to wait actually, we would like to do something like, can you please do the click and do all the actions and everything but please just stop the test there and then actually execute the next assertion or whatever else I want to do in my component. And with that we can use the helper we already seen earlier in the visit helper example. We can use directly wait in our integration test to make this happen, and what wait actually does is, it's probably a solution that we see in other programming languages framework as well, it's a pulling helper so it will actually like pull for any kind of like timed actions or any kind of like handlers still being in the event loop, and if there aren't any of them not left anymore, it will just resolve, it will return a promise, which resource, and then you will have the ability to actually execute the next line of code. So this is kind of nice, so we can use that. So let's get this into an example. And we can simply wrap our assertion at the very end into the wait helper, and then we're actually good to go. But there's one thing, like I don't know how we feel about this, it's like still feels a little bit, I don't know, not so super literal I guess, it's like a bit hard to read, like here, return wait and everything. So we can dive into some other cool helpers that are currently around there, and I can also highly recommend you to make use of, and this is ember-native-dom-helpers, and with those we actually get the really cool functionality of wait in many of the helpers out of the box, because all of them will also return a promise that we can wait for in our own tests, and therefore make it possible to create really nice readable code in our test base, and using it together with the newest feature, a wait and a sync, we have a nice way to actually make this happen. So let's see how this actually looks like in a test, so just that you have an example, we could then simply in our test case make the function callback that we pass to the test an async function, and then once we get to the point where we actually want to do the click, that actually handles the requests and actually like the async operation, we can just say wait, and this will ensure that actually our tests will first of all resolve everything that is initiated by the click, and then just like move on to the next assertion that we put in here. So this is really nice. And also really nice to notice, also notable for the other examples down the road, what you also get from ember-native-dom-helpers are like helpers like find, so you can also get like any kind of HTML elements out of your component, and in the further examples we will also use that to actually grab our elements. Just that you just say you saw that. So this is great, we have a passing test, right? But we now go like, oh now we can be more ambitious. Let's make this even cooler, let's make the loading of the comments like actually a concurrency task, right. It's really cool because with increments of the task we can do really nice things like flagging the current state of the operation that we just started, and many other cool, fancy things that we might not be able with our own implementation. So let's do that, let's actually write it, and let's also say that we have other tasks that actually will not only do this perform and actually execute this task, but also will allow us to reload it every now and then so we don't have to always click on a speech bubble to always find out, oh, the new comments and everything, we can just like let the page stand like that and actually iterate over that again and again without any issues. So we have a really high timeout, right, so we assume it's not like every two minutes, that we have like a new comment there, and we just like go through that again and again. This looks really cool, like when we implemented. And then we try to write a test for that, right. So now we don't have a click or something like that, it's just like we render the component and then actually this loop should start, right, and then later on we would, for example then wait, and that just like return the assertion. So but now we do this, and then, oh test timeout, what happened here? The problem is that because we tried to wrap all of this task into a y loop and we reload the comments again and again. Our test helper wait actually has no possibility to check, oh yeah right, we don't have any running instances anymore of any kind of timers because there is always one, right, there is always a new timer being set up in the timer that you always schedule. And this is kind of like a bit difficult, and we could be naive and we could go like, oh okay let's maybe just test this iteration once, and then the rest I don't really care about anymore, we just put like a guard check there if we're in testing mode, just like return, I don't care about that, right, I don't want to test the timeout. Then we think about it a little bit deeper, and we go like, yeah but what, I don't know, what's about the space below the guard check, right, what if someone adds new code there, if something is there? Or maybe we want to test more, right? We realise this is kind of like a really nice place for bugs and to put them just like hide them forever from our test. And also we might realise, actually testing like several iterations of the loop might sometimes be beneficial. For example, I want to check if it actually like, I don't know, happens several time, because it's really important for my functionality of my code, right? So with that, we have to think about something else. The cool thing is, if you check out the Ember concurrency documentation as well, you can find that using the run loop and also like the cancel time as method, that you can find in the run loop package of Ember, you have an easy way to actually cancel all the timers that are currently still active in your test arbitrarily, and just tell your test, okay, after a certain time, I would just say, let's just leave it here, and let's just make actually the wait helper actually find and resolve so we can actually do our assertion. And yeah, then we do that, and then we go like, okay, now we got this, it's still failing. Okay, yeah, let's have a look at this again, right? We still have the problem that we have a super long timeout in this kind of like checking and reloading of our comments, and then we go like, okay, let's just turn it up, right, let's just like put 50,000 milliseconds in there, and then we go like, oh wow, this really takes a long time, I don't know, maybe I should go back to sleep. So this obviously doesn't work, right? What we now can do is use a little bit more sensitive approach to how we utilise the Ember testing guard check in our apps and our test suites. We could for example say, okay, we don't put it directly into the actual functionality we want to test, but maybe at least we turn down interwall that we actually apply to the timeout that we put into this functionality, and so we could just like say, okay, from testing mode, just make like one millisecond or whatever, and otherwise if we are actually in our applications, we don't want to reload it so much times, we actually probably want the actual value. And once we've done that, we can actually then again turn down our timeout provided to the later call, and then we can finally actually do our assertion. And then, yeah, it finally passes. So we're quite happy. And those are pretty cool. And also what I found is that, yeah, especially using ember-native-dom-helpers, it kind of like also makes it more readable to write our test suite, and also like to read it, and if you want to learn more about where this kind of like API is influenced by, and also what kind of like other developments that already kind of like precedes, what it already hints at in the Ember development, I would totally highly recommend that you check out the grand testing unification RC, by Robert Jackson as well, because it also speaks a lot about these things. So this was already kind of like a nice example, and one thing that I found also really fun just recently that I saw was something related to time travel, so I realised that sometimes there would be tests where it's like, okay, sometimes I run that on one day, and sometimes on the other, and then they fail, what is happening here right now? And in this regard I want to also show you a short example because I found it like quite an interesting example that I just stumbled upon. So my demo app, for example, I always have, because I always need a little bit of pressure to actually get things done, I always have this kind of like reminder up here, in the upper corner telling me what kind of like day of the week it already is, right, and if it's like Friday, it goes like, okay it's release day, you have to get something done. So this is kind of like nice, and if you wanted to test that, like if it's actually there or not, you could be first of all be a little more straightforward and go like, okay, usually it's not there, right, so let's something like this wizard, the overview page, and then assert actually that it doesn't display any of the disclaimers and then the first assertion just forget about that, just like the second last and the last, and that these disclaimers actually not there. But what happens is then Monday to Wednesday obviously, they just like pass, right, Saturday, Sunday as well, but if I want to work on Thursday and Friday on this app, I have a problem. So yeah we have to think about how we actually can ensure that our test knows which day it is, and it's reliable. And the cool thing is there are different add-ons that you can actually use, and different libraries to make this happen. Like there's for example, Lowlex, which is like a part of Xenon, and yeah other libraries I would also suppose are out there, but what I would like to show you is an example with ember-cli-timecop, which usually also works all right, except for maybe some Ember versions where you might have issues with that then maybe you might want to look into Lowlex as well instead. But nevertheless, let's just have a quick look on how this example looks like. If we, on an integration test, we could for example make sure that this kind of like date's always set, and we can always make sure every hour test starts out from the same setup, so we can use them for each and after each hook, to actually make this possible, and we can instal timecop, we can then just like set a date, and then after each of our test cases, we will just like uninstall it. And once we have done that, we can just say, okay, let's go back into our test again, actually make our assertions, and now finally, yeah, we can actually make this test pass. And also a great thing is if we now want to just have a test case for Friday, we can easily do that. We can travel through time with timecop and say, okay, let's put current date on here, so it's like today, right, and then we find out, all right, yeah, today, the newsletter has to go out, right. So that's pretty cool. And with that said, I hope that you already, maybe have already seen some things, but also learned some new things about helpers, testers, add-ons that can help you to handle timing in your web apps, and I hope you get a better understanding of how timing actually matters in your tests, and with that said, I wish you a great EmberFest, and thank you very much for your attention.