Sessions is temporarily moving to YouTube, check out all our new videos here.

Progressive WebVR Apps

Perter and Diego speaking at JS Monthly London in September, 2017
Great talks, fired to your inbox đź‘Ś
No junk, no spam, just great talks. Unsubscribe any time.

About this talk

A practical demo of progressive WebVR apps, using PWA technologies.


First, I'd like to start with progressive web apps. This is a term that I imagine probably most of you've heard and are fairly familiar with. But I think it can still cause quite a bit of confusion this term, and there's still lots of different meanings that you'll hear from different people. So, what meaning will I share about it? Well, probably my favourite succinct definition for it is this: web apps upgraded to be first-class apps. Alternatively, as Alex Russell said, who came up with the concept and also named it along with Frances Berriman, they're just websites that took all the right vitamins. Another quite a good way, I think, of describing them is that it's kind of the best of the web and the best of native apps. That's what we want. The best of the web being these things that we know and love about our web platform. The fact that we can write web applications that work across any platform. They're frictionless, so people can just tap a link or type in a URL, and they can access this immediately. They're discoverable on any search engine. And they're open for other developers and also for our users. We don't have any licence fees to pay or profit shares to give away. Everything is open. So it would be nice if we could combine that with some of the best things about native apps, such as having a home screen icon that we can tap to load our app and have that display then in a standalone window, have them work reliably, independent of the network conditions, and have it load instantly or near instantly from cache. And then, these things aren't really core features of progressive web apps but you can also add to that some extra features that the web has been gaining over time like push notifications and other device features like accessing the camera. So now to share a few myths and hopefully dispel a few about progressive web apps. The first being that they're kinda like this magic recipe, right? That no one really understands. But really, they are based on a set of web standards. So you can think of PWAs really being made up of just these three things really. Firstly, you'll need to serve your site over HTTPS, you'll need a service worker with some basic caching, and you'll need a web app manifest that defines a few basic properties like a title and an icon. Then, the other stuff that you might hear that goes along with PWAs is generally optional, extra stuff like push notifications often gets talked about in context of PWAs. The other thing that isn't on here but that is also important with PWAs is performance, and that's something we'll get on to a little bit later when Diego shares an example and we'll see Lighthouse, which will score your PWA based on performance too. Next myth is that well, these are things we already have, right? These are just a set of standards that were already there with the web, so isn't this just a buzz phrase like just an amalgamation of things. But it's more than that because progressive web apps get some special features if browsers recognise these traits and these technologies that are being used as part of them. So, for example, on the left, this is Samsung Internet, and if it detects that you're viewing a PWA based on HTTPS, service worker with basic caching, web app manifest with some basic properties, then it will dynamically switch that icon that's in the top left in the URL bar, which is normally a star for bookmarking. It will switch that for a plus icon that gives you an easy way, an easy shortcut to add that to your home screen, which is nice. Then, also, depending on browser metric surrounding engagement and when they determine that it's a good time to display this to a user, they can also show up an add to home screen banner for you, so that's an extra way of helping users to know that they can actually add this to their home screen and to prompt them to do so. And in Android O, in Chrome now, we even have web APK, which means that when you do add your web app to your home screen, you'll automatically get your web application route as an actual native Android app, which means that it will show up, for example, in the actual apps list and you'll be able to access actual app settings for it. So this is truly, actually making your PWA a first-class app. And they're even coming to app stores. So Microsoft are already working on this, and it's beginning to happen in the windows inside of preview, I understand. They are doing passive ingestion of PWAs into the Windows store, which is now the Microsoft store. So unless you opt out and if your PWA passes their test and they think that it's a good enough experience for people, then they can automatically pull it into their app store. And similarly, although this hasn't happened yet, we've heard some pretty strong hints from some of the people on the Chrome team that this is something that they are looking at in terms of the Play Store as well. So, of course, it's the web, so you don't necessarily need your web apps to be in these app stores. But it can be a pretty nice thing for them to be there as well for that extra discoverability. If people are looking for apps, then maybe they're gonna be looking in these app stores. So it's quite nice that you'll surface up there too. Myth three is that PWAs are only. Oh, skipped ahead. They're only really useful if you're interested in making an offline application, and if you don't have any particular requirements around offline usage, then maybe you don't really need to worry about it. But actually, the real thing about progressive web apps and service workers and caching is that it gives you reliability to have your site, your app work independent of those network conditions. And you don't have to worry about li-fi and being in Clapham Junction and not having a signal. And everyone needs reliability. That's what PWAs can give us. Myth four, we do see this sometimes from some tech publications. PWAs are a Google brand, or they're a Google only thing, or they're Google PWAs. But really, they're a cross-browser collaboration as demonstrated here nicely at the PWA Dev Summit, which was in Amsterdam last year when some of our Samsung colleagues were on stage with Google and Mozilla and Microsoft and Opera. And so, yeah, it's just like web standards themselves are a cross-browser collaboration, progressive web apps are too. Myth 5, PWAs are not ready yet because they're not on IOS, right? Although, we did have the good news recently that Apple have actually put service workers officially in development in Safari. So it's progressing, but you might think of this as, "Well, they're not on IOS, so they're not really here yet." Well actually, a whole bunch of organisations, big organisations like these and many others are embracing PWAs and making their sites PWAs already or making new sites that are PWAs. You might have come across some of these. So people are doing it now, and you can do this quite safely with progressive enhancement. You can have your app, your site work just as it would do normally on IOS and on Android or other platforms where it has support for these extra features, you can use those and make the most of them. The last myth is that PWAs are just a mobile thing. So it's only for mobile apps. But actually, we will see that they are also for desktop. For example, so this is Samsung Internet on DeX, which is our little docking station that you can plug your keyboard or mouse into and turn yourmobile phone into a desktop experience. And this is showing that you have PWAs on desktop too. So just like in the mobile version of Samsung Internet, you have a little plus icon there if it's a PWA and you can do add to home screen. And so in this case, add to home screen adds a desktop icon, and when you load it from there, you have a standalone application like a native app. Let's just go back to present. Also, desktop, these kind of features are coming to Chrome OS and also Edge as well. And not just mobile and desktop but virtual reality too, and that's what we're going to be sharing a bit more about today. Because if you are developing a virtual reality web application, then why wouldn't you also want to take advantage of some of these great features of progressive web apps too? So as Dan Scott shared, he wrote a blog post about this. What if I could just escape into my lovely virtual reality scene wherever I am, without worrying about connectivity or how long it would take to download? So this brings us on to progressive webVR apps. And at this point, I'm gonna do a quick microphone switch here with Diego. So Peter has given us a great introduction into progressive web apps. What we're going to see now is a special demo that we prepared for today, unlike the one that we had last JS Monthly where we had a bit of an issue with illegal panda cloning that happened on stage. Right now, we needed a PWA that was iconic. We actually looked for many hours, what was the PWA that was going to make the cut? And of course, I don't know how many of you are familiar with this logo over here. Can I see some hands? Yes, the famous airhorner by Paul Kinlan. Well, he is already informed and he actually approves the webVR version of the airhorner. So not going to spoil it. I think you already know what's coming, but the meetup is going to get a bit loud. What we want to do is we want to take the airhorner, and we want to upgrade it to the next generation of the web, which is pretty much VR. So the past probably day and a half, we were working very, very hard in order to get to you the airhorner. If you all want to go to the link and see the airhorner, you are going to be able to see how it has evolved. Probably you're not gonna see many changes at first, but just wait and see because it is magical. This changes absolutely everything about PWAs again. So what we're doing is pretty much that we are using PWA technologies, which Peter has explained, to cache audio, the very famous audio from the airhorn. We are using a splash screen to load or play audio. This is basically a trick that we're doing because we need the application to be able to play audio. And in some browsers, like mobile browsers, you actually need to re-gesture user input first. And then we are attaching animations to event. So when you load the application, once you go into start, you're going to run into a webpage that looks more or less like the initial airhorn, except then it goes crazy 3D, and it's magical. What we are having is again all the benefits that Peter described like Yeah, there you go. That's what I wanted to hear. Great, thanks. So we have an icon on the home screen. We have the application per se, and we actually tested that this application works in different types of browsers, including VR browsers. Here we have a screenshot of how the webpage looks inside the Samsung Gear VR browser. And I've actually loaded here. I'm just gonna go in VR and go into the start and the immersive mode. Now, I'm just gonna give the headset to some lucky person in the audience, and feel free to tap away. This was something that we did in the course of a day and a half. We did run this earlier today like two hours ago on Lighthouse to see if we were not failing epically, and it turns out that we scored quite well on the progressive web app. Now, my website is not automatically redirecting to HTTPS, so they are things that, I know, shame, right? A bit of shame. I know, and I'm sorry. I'm very sorry for that. But we can see that the part that was most impacted was performance. Now, here has to do with the script, the A-frame is kinda like stopping de-rendering because it needs to, of course, create the 3D elements, et cetera, et cetera. But nonetheless, I think it's quite good. Yes? Can we get some approval from you? Amazing. Great. Okay, so now that we've seen a basic example of how we've taken a webVR experience that's running on, of course, on the browser. Then, we decided to test that okay it needs to run in mobile phones, and it is running in mobile phones. You've seen it in your devices when you actually started tapping. We tested it in three degrees of freedom devices, and if anyone wants to try it out in VR. It's quite basic, but if anyone wants to see that it actually works, please feel free to ask Peter for the headset. We have it running on a three degrees of freedom or a just orientation with the tapping here. Before coming here, we actually tested it on an Oculus Rift, and it is actually working. The problem is that we didn't implement anything to actually tap. You know that when you have the Gear VR, you can tap, and it represents a click. When you go on the phone, if you tap the screen, it represents a click. No worries. On the Oculus Rift, we actually needed to create this click event because it doesn't have any touch pad or anything that triggers the event. Then again, it's probably one of the tips that Peter is going to show us following, which would be related to progressive enhancement and making sure that an experience works across different types of devices. So that will be for the airhorner VR 2.0 version next reloaded. So that's it for the example. The code is already on GitHub. We are going to share the code and the slides, and Peter is going to give us some tips on how can we mix PWA technology with webVR to create these type of very engaging and amazing experiences. We're coming to the end of the presentation, but I wanted to share a few more thoughts about the progressive part of these progressive webVR apps. Starting with a great video by Arturo, if this will load, because I think this really shows that while, and some of you might have seen this animation already about how you can have these experiences that work as normal in normal browsers and mobile devices as we're used to. You can also have what we call a magic window effect, where you're using the accelerometer of the device, and you can see different parts of the 3D scene by moving it around. And you can have this going up into the higher-end devices, and it works then if devices like a Daydream and the Samsung Gear VR, and then even going up into the higher-end devices, which it cut off at the end, but things like the Rift and the Vive. And so the idea is that you should really focus on that initial mobile, low-end device and then progressively enhance, build up more functionality for those other devices too. So virtual reality is not an excuse to make people wait. You might think that while people are gonna love my VR experience, my VR airhorner, I'm sure people would wait for that, but maybe not that long. But really, you're still not going to have very long before people give up and go, "I'll try something else or come back to it later." And then never do. So it's best to get things on screen, get the first part of the experience up in front of people as soon as possible. One way you can do this is there's a model of Call of Duty assets demo from PlayCanvas. Hopefully, this slide doesn't look too threatening. Sorry about that. This is not saying, "Downloader higher res as this post loads." Sorry. But it's to share their example where they are initially loading low-res assets, low-res textures, so you get to see this model as soon as possible, and then once it's loaded up, it's then downloading, streaming down the higher-resolution textures and then it overwrites them, replaces those on the fly, which means that as you're viewing this, probably by the time you then start to zoom in and look around this model, you probably have by then got the higher-res assets in anyway, and then you'll be able to see the higher-res version. We also like to talk about this concept of continuous browsing experiences because you don't necessarily have to just think of your app as being a VR app. It can be a mobile app, which has a VR element to it. So you could imagine that people are viewing your page or your application on their mobile phone and then maybe there's some part of that says, "Hey, there's actually a virtual reality part "of this experience that you can try." So that they go, "Okay, I'd like to do that." So they slot it in their virtual reality headset and in Samsung Internet that will then make it load automatically in the Samsung Internet for Gear VR in the virtual reality browser. Then they can experience their virtual reality version. And then, once they finish, they can take it out, and they're back to the mobile experience again. As an example, sorry. We can turn the sound down, now couldn't we? Thank you. So, as an example, imagine you were reading an article about the Nasdaq stock price and whether there might be another bubble again. But it's kind of, you could read about this, but you might not really feel it. You might not really experience it. So imagine if then you try out the Nasdaq rollercoaster, which lets you actually take a ride on this stock price and then you actually really get to feel the ups and down, right? And it was an experiment by the Wall Street Journal, and it was more just to try out a concept. But they had a really good reaction to this. People were saying things like, "I never would've bothered to read an article about this, "but this really brought it to life for me." This is the scary bit when it, ah! It's also important to remember that in these virtual reality browsers, our 2D sites, our applications that we have already, they'll still continue to work as they are because these VR browsers, even though you're in a 360 environment, you have a web-browser window. You'll have 2D essentially window that you can still view your content as normal. You're just getting me back for before. And you can use the virtual keyboard, or you can use voice input even to interact with it. Then finally, it's interesting to think about this third axis itself as being a kind of progressive enhancement. So imagine if you had your regular website and you had your HTML and CSS, and then just by adding some extra CSS 3D properties, you could then take advantage of that third dimension. This is a concept that has been talked about a lot, and there's been some initial work gone into it. But at the moment, this is not something that's available in the web, at the moment, to do webVR apps. We need to have a separate webGL canvas. Although with libraries like A-frame, that's starting to get as more towards this really nice progressive enhancement model because it is actually letting you write HTML. But hopefully, in the future, we might also have this CSS 3D enhancement and be able to really easily progressively enhance our sites to take advantage of 3D. So with some of these things, it is kind of early days for virtual reality on the web, but we are progressing.Sorry. I think at that point I probably should just end this and say thank you very much.