Sessions is temporarily moving to YouTube, check out all our new videos here.

Cavy, A New End-To-End Testing Framework For React Native

David and Abigail speaking at React Native London in June, 2017
Great talks, fired to your inbox 👌
No junk, no spam, just great talks. Unsubscribe any time.

About this talk

Cavy is an open-source end-to-end integration test framework for React Native, developed for use with both iOS and Android applications. Learn how Cavy works, plus a live code example against a real React Native app.


Hi everyone. Before we start talking, just a quick show of hands. I'm assuming that most people here have written a little bit of React Native, 'cause this is a React Native meetup. But hands up if you've written tests for a React Native app. That's better than when we spoke to the React people at Facebook. Hands up if you enjoyed that testing experience. Okay. Slightly less, good. And hands up who thinks that testing is just a big waste of time anyway? Okay, just a couple. That's good. Okay, well, we're gonna talk to you a little bit about a tool that we built for end-to-end testing React Native apps. And we called it Cavy. And we think it's pretty cool. Hopefully by the end of this, you will as well. We're gonna tell you a little bit about what Cavy is, why we think that you need it. A little bit about how it works, and sorta how we built it. I have the lucky job of doing the live coding demo, if everything goes okay. And then we'll also talk a little bit about where it's going in the future. But for now, I'm gonna hand over to Abigail to talk a little bit about what Cavy is. - So, what is Cavy? This is Cavy. So, before you ask, he's not a bear. He's supposed to be a capybara. But capybaras are really, really hard to draw, so. I'm a developer. According to Wikipedia, a cavy is a family of rodents native to South America, including the domestic guinea pig, wild cavies, and the capybara. Wikipedia then goes on to say that many cavies are promiscuous, forming no long-lasting social groups, although in some species, males maintain harems of two or more females. This guy. How unexpected. The other end of the scale, Cavy is also an end-to-end testing framework for React Native. Cavy was born out of our strong belief that there must be a simple and clean solution to testing multiple React Native components. Cavy is open source. It's end-to-end. And it doesn't touch any native code. So, because of that, it doesn't touch any native code, it runs on both Android and iOS exactly the same. You set it up once. You write the same tests for both, and it runs exactly the same. So it fits in really well with the React Native way of doing things. Cavy also runs tests within your live application. So you can run them on your iOS simulator, or on your trusty Android phone. Yay, Android. So why do we need Cavy? Aren't there already enough testing frameworks out there already? So I just wanna talk a little bit about why you might to consider adding Cavy to your testing setup. So just setting the scene a bit. You are all mobile app developers. You're coding in JavaScript using React Native. Loving life. You got a new feature and it's ready to go, and everything's tested. So you've got your unit testing. You're maybe using Jest and Enzyme. And you've even got a little bit of snapshot testing in there too. Which is something we'll go on to a little bit later. So you're feeling pretty confident. It's build time. But oh no. There's fire. And there's fire everywhere. So your components may have looked like they were working on their own, and the new feature suddenly looked like it did to you. But somewhere along the line, something's gone awry. Your key user journey has been disrupted. Shame on you guys. So what went wrong? Let's start with what went right. So, you've had a unit testing all set up with Jest and Enzyme. So how many of you are familiar with using Jest? Yeah. As I thought. Jest is kind of the well-trodden route for unit testing. It's the recommended unit testing framework for React Native. So for those of you who haven't used it before, Jest comes with um, full mocking support, a test runner, assertion library. And it also lets your do snapshot testing. So, snapshot testing is where you take an image of your app or a decent representation, in a specified state. You compare that to a snapshot taken earlier. And if there's any differences, your test fails. You can decide whether it's failed because something's gone wrong or whether that's a normal deviation. So there's also Enzyme, which a company you've probably used as well. Enzyme is another JavaScript testing framework. And you can use that alongside Jest. It gives you the ability to shallow render part of your component tree. And just test whether your components are rendering correctly. So again, it's a good tool for unit testing the basic components of your app. So coming onto end-to-end testing. We kinda found that the, while the decision to use Jest and Enzyme was quite an easy one for us, deciding what to test end-to-end with was a lot more trickier. So there are a couple of different options out there. And the first one that we kind of entertained was using Appium. And we found that kind of the overhead to set up Appium and integrate it into our app was really big. It felt like it was quite a lot of work to get it in there. And, work, no. So, Appium also uses native hooks to access your components. So it doesn't work the same with Android and iOS. So yeah, that was kind of like a no for us. Moving on to Calabash and Cucumber. It's another setup that's out there for you, but it's quite similar to Appium. It uses native hooks to access your code, through accessibility IDs. And you have to set up Calabash IO. You then have to set up Calabash Android. And that's just the thing. We didn't really feel like you should have to do that in React Native. So, not finding anything that we were particularly happy with, we felt that there was a gap in the React Native testing ecosystem. We gave ourself a week of dev-time. And then Cavy was born. So we think Cavy is pretty great, and a couple of other people do too. And actually, we do know of other companies that are using Cavy in their production code. Which is really cool. So just before we launch head-on into the demo, I want to just give a little bit of background about how Cavy is working behind the scenes. And actually it's quite simple. There's just kind of three main components. And the first one of those is the test hooks. Or what I'll call test hooks going forward. So, the test hooks are what you attach to your components that you are going to want to manipulate and access in your tests. And, to do this, Cavy provides you with a ref-generating function. And here it's the generateTestHook that you can see in the highlighted section. And basically, to hook in your components that you want to test, you just pass this in as ref. You can see that the function takes a string identifier. So here I just called it Scene.Input. But you can call that whatever you want. It's just the string that you'll then use in your tests to refer to that component. So we'll refer to the text input as Scene.Input in this way. So once you've hooked up your component, Cavy then stores this in what we call the test hook store. And that's just simply a store of all of the components together with their identifiers that Cavy can use to store the components you want to test. So the second part is Cavy's helper functions. And you can see that they're quite simple right now. It's quite bare bones. You got fillIn, press, and exists in this situation. And for anyone that's ever used Jasmine or Ospec or Mocker or any one of those things, these are gonna be quite familiar to you. As is the way that the kinda spec file is built up. So in this case you can see that we're describing My feature, and we're checking that it works by filling in a text input, pressing a button, and checking that the next scene exists. So it's just testing a basic transition between scenes. We've kinda just included functions at the moment that we found we needed when we're testing our own app. But it's quite easy if you wanted to add your own custom helper functions, to suit your own needs. So thirdly, there is the app wrapper component. And this is the kind of most abstract component in Cavy. So you can see here that we're importing, at the top line, Tester from Cavy. And this tester component wraps around your entire app at the higher most level. So in the highlighted section you can see the tester component wrapping around my app, which is our app. And to summarise the most important things that this app wrapper does for you. It makes the test hook store available to you. So here we're importing the test hook store from Cavy. And you pass this in as a prop to the Tester component. So you've got Tester, and store is the testHookStore. It also actually runs your specs when your app boots. So we're importing your app spec from wherever you're storing them on the second line, and then you pass these in as an array of specs into your tester component. The app wrapper does a multitude of other things kinda behind the scenes. But the most important one is, well that I can explain well, is it re-renders your app between each test. So this makes sure that tests aren't, kind of um, don't rely on other tests having been run. Or other tests passing. It makes all of your tests independent of each other. So, without further ado, it's demo time. - So we've got a little demo up here. It's just kind of like an employee list, so to speak. You can see a sort of scrollable list of people. You can search at the top to filter it down. You can tap on a particular person. And you get a little bit more info about them. And also some buttons to email them or call them. And this is cool. Sort of. But we want to add Cavy to this. So the first thing we need to do is we need to add the Cavy library. So that's pretty easy. We can just do yarn add Cavy. And because it's a test thing, you can add it as a dev dependency. npm does work as well, but obviously we're using Yarn. And then, we don't need to run React Native Link or anything like that. Like this is all JavaScript. There's no native part of this at all. And then the next thing is that we need to wrap that, our app in that tester component that Abigail mentioned. Just wanna quickly run through, just in case anyone here isn't super familiar with React Native app structure. And also, I'm mindful this app is a little bit old. We've got our native folders, Android and iOS, which we don't need for this. We've got our app folder, which has got our app JavaScript in. And then down here somewhere we've also got, we've got a leftover index file, lemme just get rid of that. That doesn't do anything. Cool. We've also got our two entry files, index.iOS and index.Android. But all those are doing are actually just importing our main um, file, which is where the app starts. And this is where we need to add that tester component. So please say something if I make a typo. But we're gonna start off by importing that tester, and also our test hook store from Cavy. We're gonna initialise a new one of those test hook stores. Nothing so far. And then because there isn't actually a React component in here yet, so I'm also just gonna import in React. Okay, and now we need to write that app wrapper component. And all this is gonna do, as Abigail showed in her example, it's just going to return that tester, which takes a whole, it takes a bunch of props that you can use. We're gonna use the specs one, which is the array of your test functions. The instance of that testHookStore. And also I'm gonna add an optional one here, which is wait time. And that's just kinda for demo purposes. It's just going to pause. Cavy's just gonna wait between each test that it runs. So that we can just see that it's doing something. Otherwise it kinda happens a bit quickly. And then finally, all this is gonna do is render our original employee directory app. And then we just need to change our app registry line here to register that app wrapper instead of the employee directory app. So we'll just check to make sure everything's still working. Which it is, cool. But we can see Cavy isn't doing anything yet. And that's because we haven't written any tests. But before we even write some tests, we need to hook up those components that we want to interact with. And actually, the first thing we need to do is decide what we're gonna test. So I think we'll probably test that you can search. And that it filters that list. And also we'll test that when you tap on someone, you get transitioned to a scene where you can um, you can email them. And there's an email button on there. So if we start with that search, the text input on the search bar. So if we go and find that file. Which is this one, the SearchBar. And then, the first thing we need to do is import that hook helper function from Cavy. And then I'm gonna add one of those ref hooks into the text input. So I'm gonna add a ref here, this props. generateTestHook. And as Abigail said, you can pick any naming convention here. It's just a string at the moment. You can't have two that have the same string. That's not because it's impossible, but just because we haven't done that. So you can really kinda do what you want here. And then we just need to change it so that instead of exporting this component, we need to export our wrapped component. So export default hook. And then, we also need to wrap our email button and also the list item as well. So that we can tap on one of those, we can get Cavy to tap on one of those list items. I'm not gonna type all of that out because it is too much effort. And it's just gonna slow us down. And I'm gonna make a mistake. So I'm just gonna swap to a tag here. So we can just go and have a quick look at those. So it was the email button, which is in the action bar. So we can see that there's a, there's a test hook in there. And then also the employee list item. So for the employee list item, I've kinda cheated for the purpose of this demo a little bit. What I've done is I've, because we can't have two components with the same identifier, because Cavy won't know which one to interact with. I've just made a synthetic sort of identifier from the first name and last name of the employee. So that I can refer to them individually. You could change this to, you could enhance Cavy to make it work with multiple components with the same name. Or you could tell it to only hook in the first one or something like that. It's kind of up to you. So now we've got all of our components hooked up, we can write our specs. So I'm gonna make a specs folder. And I'm gonna make a file. And I'm gonna call it EmployeeListSpec. So, doo-doo-doo. Cool. So, the specs are just functions. They take a spec argument. And that's what gives you all of those helper functions. As Abigail said, this structure will be very familiar if you've done anything with Jasmine or Ospec. Or anything like that. And so I'm gonna say, I'm gonna do that first one. So the um, when you type in the search input, it filters the list. So listing the employees, it filters the list by search input. So all of these, as you may have noticed on the slide earlier, these are all, ooh, nearly lost a bracket. These are all async and await functions. And that's because, if you've ever used like um, headless browser testing, if you've done something with maybe Capybara, hence where the name Cavy comes from, or um, anything like that with PhantomJS and things like that, you know that they all kind of wait. Like if you ask them to say like um, check to see if there's this text on the page, or click this button, or do this, they all wait because you may have an asynchronous action or you may have something going on in the DOM. It's very similar here. So all of these functions all wait for a period of time, which you can customise, to make sure that, you know, the button is rendered, or the scene has transitioned or whatever it is. And to avoid all the callback soup, we're obviously gonna use async and await. So I'm gonna say. I'm gonna check to see first of all that an employee list item for someone that we're going to filter away, is present. And then I'm going to fill in that search bar text input with a string, oh, nearly. And then I'm gonna check that that component has now gone away. And then I'm gonna check that the one for Amy is still there. Cool. So now we've written this. And I think it looks okay. I missed a semi-colon but, y'know, I'm not counting them. But now we need to add it back to our, our index file. And we need to add it to that list. Sorry, our main.js. And we need to add it to that list of specs, that array of specs. So the first thing I need to do is just import that function that I just wrote. So import EmployeeListSpec from the specs file. And then I can add it to this array. Cool. So hopefully, if I refresh this and nothing has gone wrong, we can see that, and it happens quite quickly, so we'll do it a couple of times. But you can see that Cavy's starting, it's finding, it's loading up that first spec. It's running it and you can see what it's doing. It's loading, it's adding something into the search, and then it's checking to see that the element is still on the, in the scene. And then it gives us a nice big tick, which is good. So I'm gonna write a second one as well. Just for the fun of it. I'm gonna do that one testing to see that it can transition to another scene, and that there's an email button on that scene. So tapping on an employee. It shows a button to email them. So I'm gonna, I'm gonna fill in the search again. C'mon. Ugh, it's this keyboard. I'm not used to it. And then I'm gonna press on one of those, on that uh, list item. And I'm gonna add, there's a helper function we can use called Pause. And what that does is that just pauses for two seconds. And that just allows the scene to transition, so we can visually see that it's working okay. And then I'm gonna check that that email button is present. Okay. Does that look okay? Yeah, okay. So now if I hit refresh again, we can see that it runs the first test, it resets the app, and then it also runs the second spec. And just to prove that I'm not making this up, if I make a typo here, which is not really a good example of a failing test, but we can see that it runs the first test, but then the second test fails. So I can make that pass again. And I'll just run that again. So it works through those two things, and then, hoorah, all our tests pass. We are successful developers. So I kind of flew through that a little bit. But just to kind of recap. In 10 minutes or so, we've added Cavy to our app. We've added that tester component, the test hook store. We've hooked up all of our components. And we've written a couple of test examples. And although I don't have it here for proof, this would also work on Android as well. Because, as you've seen, it's all JavaScript. And this is, we've done all of this despite the fact that actually Cavy's still pretty bare bones. There's not a huge amount to it. We have a bunch of ideas for making it better and improving it. And actually I'm gonna now hand back over to Abigail to talk about that. - Where is Cavy going next? As I mentioned before, there are a couple of companies that are using Cavy in production. So that kind of influences where we want Cavy to go in the near future. Kind of informs our decisions from now on. But we really want to be able to support continuous integration with production testing setups in mind. So as you saw in the demo, at the moment Cavy simply just logs out the test results to the console. Which isn't super useful for continuous integration purposes. So things like Jenkins and CircleCI. So in order to support this, what we really want to do is add a reporting API to Cavy. We kind of envisioned writing separate adapters that would then take that generic test report from Cavy and then send them to CI tours in whatever the required format is for each of those. So basically we're imagining things like Cavy-Jenkins, and Cavy-Circle, as packages to be developed in the near future. And that is something that we kind of want to reach out to the community for. To kinda help with that, so, poll request welcome, guys. Help. And that is about it. Does anybody have any questions? - How do you test Cavy? Um. - Yeah, about that. - We try it? That's a good question. We don't particularly, apart from just having our sample app that we run and make sure that it. You know, we have a sample app preloaded. It's basically the same as the one I just showed you. To make sure that that in itself works. So if Cavy is working in a situation in an app, then it itself is kind of working. If there's a bug in Cavy, and the app hasn't changed, then the test will fail or something will go wrong. So that would be kind of, that's kind of how we tested, I guess. Um, does that answer the question from the guy at the back? Good, okay. Um, does Cavy play nicely with Redux? Yeah, it does. We've had a few people say that, there's been some issues, so, as you probably guessed is, if you've used Redux before, the way that the store works is kind of very similar. So it uses the context to sort of emerge that store anywhere in your app. That does work alongside Redux as well. We've heard of people using it with Redux as well so, it should work fine. Do the Cavy components affect snapshot and unit tests? That's a good question. - We haven't really given that a go. - Yeah, we've not tried it so. - It's an interesting one, though. - Yeah, try it and let us know, I guess. In theory it shouldn't. - Yeah. - Do we need async / await to use Cavy? You could write those spec. Well, Cavy itself uses async and await underneath. So you would have to transpile Cavy to YES5. And then, you would then be able to use callbacks, I guess. I'm not sure why you would want to because presumably it's React Native, and you've got the package of building your code. So that's converting all of your code anyway. Is there a reason why? For the anonymous person that asked that? Want to follow up with anything? Okay, it's interesting how we get anonymous questions. And does Cavy also work with React? We haven't tried it. In theory, yes. You could use it to test HTML. It's kind of, it feels like maybe if you're thinking about end-to-end testing, you kind of probably wanna be higher up than that. You probably wanna be testing it using a, you know, a browser. You wanna be clicking on real things. The reason that we built Cavy was because there isn't really something ideal that's easy to use that fills that. So we're kind of a little bit lower than you'd like to be with sort of in React and in React Native. So really, you kind of wanna be a little bit further up. How does it compare to Wix Detox? Where's my prepared answer to that question? - It's on the side. - I noticed also that Wix was mentioned at the start. And that they're working on Detox, so. It's interesting, actually, we didn't come across Detox when we were building Cavy. But, and Detox, where's my notes? Okay. I remember. So Detox uses a lot of native code. And I think it's only iOS right now. I don't know. The person that asked this, if they know if it works with Android as well. The last time we looked at it, it was iOS only. It's just, Cavy's all JavaScript, right? You saw that you set it up with Yarn add. And that's it, like, there's a little bit of boiler plate code, but that's basically it. So it's simpler. I think that Detox, we would like Cavy to work a little bit like Detox, in that we would like there to be better support for accessibility identifiers in React Native. So that we wouldn't have to have this weird store and sort of hooks and using refs and stuff like that. Like, it's kind of a hack, right? Like it looks like a hack. Because it is. But that's because there's not really anything else that you can use to actually get to a component deeply nested in your component tree. So we just kinda bypassed that by building a store, and using refs. So ideally we would like React Native to have better support. I think there's actually an open core request for proper support for accessibility helpers and exposing that to your JavaScript. So that you could then wrap it and access those directly. Why can't Cavy wait for a promise rather than an arbitrary timeout? For example, in combination with Redux? It could, I guess. Yeah, there's no reason why not. It does mean that you're putting your end-to-end testing a little bit low level. You're sort of bringing it in, it's got a little bit more knowledge about your your state and your data and how you built your app. Whereas doing it just by interacting directly with the components, by calling their props, and you know, not really knowing what's going on underneath is probably a good thing. But yeah, maybe it could. If you've got an idea for how to implement that, that would be great. Who designed the logo? Abigail did the logo. - So glad someone asked. It's me! - Yeah, that was Abigail. - First comment was, "Nice bear." - Is there a way to use Cavy without leaking it into your production bundle? That is a really good question. And the answer at the moment is no. I think that there may be someone in the community working on adding like a, you really want maybe, like a transpiler, that rips out all of the hooks and stuff when you're compiling for production. And then that way you could kind of get rid of all that code. It was a concern for us as well. And it's a concern for a lot of people. It's like you've got your app, and then, around it you've got Cavy. One person said to me, before like, how can we trust that your code is okay? And it's like, well, it's open source. Like you can read it. And you also wrap your app in Redux. Like, why do you trust Redux? So, there isn't right now. It would be really nice if there was. It does need better configuration. It needs to understand better when am I running a test? When am I in production? You know, disable myself, don't do anything. At the moment, we kinda leave that up to you. You need to add some configuration around it to do nothing if you're, if it's in production, or whatever it is you wanna do. Great work! Why do I have to change my code to be able to test it? Yeah, it's basically the same thing, right? Like, there's a balance between what we would really like, which is, you know, you want this to be more isolated from your code. We want to be better accessed in JavaScript to help us to let you do this. The reason why you have to change your code is because that is not available. And so at the moment, you have to kind of bypass it and do something to be able to interact with those components that are all the way down in your component tree in your app. So yeah. What is needed to make it headless? I think, that's, I don't know. We don't know. It does need to like, this is a problem with testing mobile apps in general, right? Like how do you test an iOS app without a simulator? How do you test it on a Linux server? That's probably not something that Cavy is going to solve. And also, because it's end-to-end testing, it does need to run the app. That's the difference between it and something like Enzyme or Jest or whatever. That they can run, you know, in the command line. - [Man] Thank you. - Any more? Cool, thank you very much.