Sessions is temporarily moving to YouTube, check out all our new videos here.

Reimagining Reality with Microsoft HoloLens

Becky Jones speaking at TECH(K)NOW Day in March, 2017
Great talks, fired to your inbox 👌
No junk, no spam, just great talks. Unsubscribe any time.

About this talk

We’ll explore the potential of mixed reality devices like Microsoft HoloLens and identify industries that could be disrupted by this technology. Covering the current state of the art with a live demo using the latest HoloLens development edition, you’ll discover who’s betting big in this field, along with the challenges the technology faces before mainstream adoption, and learn how you can start developing for mixed reality too.


- Hi, happy International Women's Day. It's such pleasure to be here. I'm Becky Jones and I'm the founder of a company called Bektio. I'm a developer and I work creating experiences for mixed reality. Now you might've had terms like virtual reality, augmented reality, mixed reality. But before I start talking about realities, I think it's worth actually asking the question: What is reality? Sort of seems a bitof a strange question. But if we're gonna talk about mixed reality, it's important to ask what this is. Reality is the state of things as they actually exist. And so the physical world around us, we actually sense this using our senses, so our eyes, our ears. Our brains then take in all of thse signals and then come up with what our reality is and what our view of the physical world around us is. Now these days we don't just exist in the physical world, we also seem to have this sort of digital online presence as well. We have lots of digital assets. Nobody really buys CDs anymore. You tend to have digital things that you own, these sort of intangible assets. Wouldn't it be really cool to kind of bring those into the world around you as part of like your physical world? So mixing your digital and physical world is quite an interesting idea. I like to kind of think about wouldn't it be really cool if I sat at home in my living room and instead of watching like Coldplay at Glastonbury on a TV screen, what if I actually could have Coldplay in my room with me and they're singing to me and the people around me can also see them. But it's not just I'm having to put on a headset and I'm completely isolated, it's something that's a shared experience and they're working around my world. There is this interplay between the digital objects in the digital virtual version of Chris Martin and the actual world that is around me. So this kind of mixing of blending of the real world with digital objects it's kind of the space that I'm really interested in. But you kind of need like a headset or a device to view this with which has this additional understanding as well of the 3D-ness around you because we are, as people, 3D. We walk around a 3D space and so it needs to also know about the 3D-ness around it. So there's devices out there. Microsoft have come up with this device called the Microsoft HoloLens where you can put it on and you can see these digital objects blended with your real world. You might have seen virtual-reality headsets and you put those on and you're completely immersed and you're looking at a screen and you can't see the world around you. But with this, it's almost like a pair of glasses. So the optics are really interesting. You can see through it. It looks just like a pair of glasses. But if you look really close, it's like on the actual device, you can see the red, green, and blue. They fire extra light and they bounce it back into your eyes and they have extra light to sort of help you see the world around you put these additional digital objects. And then there's this call-center system which they have a whole bunch of depth sensors. This is the understanding of the 3D-ness around you that it does what it's known as spatial mapping. So an example of this is over here on the far corner. So it's actually you're moving around the room. It is literally building a map of the things around you and it's the creating of this mesh and then you can interact with this mesh. And you can place objects with world coordinates, walk away, come back to it and it still there. You can walk around it and go elsewhere and then come back to your room, and because it recognises your room and it recognises where that is in the world, it will remember. So my room is cluttered at home with lots of digital objects all over it where I've forgotten to close them down. And then they also have a spatial sound. Because let's face it, you don't just see things, you also hear them. So with the system, you've got these little red microphones here and they actually throw sound out, so it sounds like something is behind you. So this is really useful for user-interaction design because when you're kind of getting people to move around the space and like know there's an object over there, you can kind of make a noise and you kind of turn to see it. So this is kind of the setup that you really need for enabling mixed reality. So then what ends up happening is you can then start blending these digital objects with the real world around you. So they're terming them like holograms and this is an example of maybe an architectural practise where they would want to see the models that they have. Because for years we've kind of been locked into this 2D world. Many of us create 3D things, whether it's 3D games. Product designers are often creating CAD models and yet they're locked into this 2D world of accessing it via the screen and building it via the screen. But what if you could actually... Sorry. What if you could actually come out of that and then you build it in a 3D environment? You can build it. You're doing your CAD model of a car and then you can throw it out in front of you and then wander around it. How could we control this kind of mixed reality? What is the ways that we can interact with that? Because let's face it, we're all used to using keyboards, and mice, and we swipe on our smart phones now. What's the kind of system that you might use to sort of interact with mixed reality? And so, these are actually like relativesof mine. I took it to an 85th birthday party for my grandma and have everybody trying out. That's my dad on the far side. He was the one who actually told me about many, many years ago, he said to me... We're watching these music videos 'cause we're really We used to record Top of the Pops a lot and he actually said to me, "Yeah, one day, that'll be in the room with you," and so that's why I started with that because my dad sort of inspired me to get interested in this. But anyway, in terms of new ways to interact, the way that you would navigate the space and move things around, you gaze, so you move your head. At the moment, it's purely on head control. There might be future versions where there's like eye tracking, but for the moment, you move your head, so look at the object you're interested in. Then you put your hand up and it recognises your hand. So there's a number of gestures that Microsoft have defined. So they've got the air tap which is kind of your equivalent of a touch to select and then you might kind of scroll to move things around and they have this gesture which is the bloom gesture and that will close an application or start up the menu. So when you see me doing the demo later, you'll see me doing all this kind of stuff and you'll know what I'm doing. And then the other way of controlling it is also with voice. So you can kind of look at things and say select or you can kind of ask Cortana to do things because this device actually is a full Windows 10 device. It's a complete computer. So with Windows 10 comes Cortana, so she's in here too. So you can ask her to find you information and open different applications and record the things that you are looking at. So it's all very cool, me talking about having Coldplay in my living room. But what are the real-world actual applications of this? As I was saying in terms of creation and design and if you're creating this kind of CAD models, wouldn't it be cool to actually be able to see it in real life? Assembly and manufacturing could be another industry to look at, obviously the entertainment industry, I mean with Coldplay, but also again, you're no longer just confined to a room space which you've predefined like you might've seen that in virtual reality headsets where they have these kind of additional sensors that work out where you are in the room. This can kind of do it all by itself. It has this what it's known as inside-out tracking. So the whole world is your play space, right? So you can create some really interesting games with that. Training and development is another one. I don't know about you guys, but my undergrad degree was in biochemistry and I spent a lot of time trying to think about 3D concepts but by reading them and sort of I'm looking at 2D images. Wouldn't it be so cool to explore medical concepts by actually seeing MRI scans in 3D that you can wander around? And then they kind of demo-d a Skype application as well which is quite an interesting field. Could this shake up communication? All right. So one example that I just want to show you very quickly of an industry partner that I've been working with Microsoft. Let's make this. Oh no, don't. We'll click that. Oops. Turn that down. So this is actually, oopsy daisy. So this is Caterpillar. What they've actually gone and done is they're using a Vuforia to recognise the different images within a catalogue, different things, and this product is a particular one that she was looking at before. She then decided to sort of make it full-size so she can look around it. Then you can kind of modify and look at different options. Really great for salespeople for Caterpillar I imagine because it would be a bit hefty to sort of cart the actual thing around with you to sell. So Oki-doki. Let's just go back to the presentation, wherever it's gone. Okay. So there's also opportunity for collaborative design as well. So a couple of people could have a headset. This is an example they've done with AutoCAD where two people are looking at the same thing, one is making a change, the other can see the change updated in real time and then they can also make changes. And so this I think it just blows my mind in the terms of like what you could do with collaborative design. On site engine maintenance is I think, again, this is another one from Caterpillar and Vuforia. Oki doki. I'm gonna turn that sound down. One more than this. So again, just imagine, it doesn't have to be Caterpillar. It could be any situation where you would send somebody on site to maintain something and you might want some additional information so to help the with troubleshooting about the thing that you're trying to solve. So with this, they were actually pulling in some sensor information from the part that was faulty, so doing a little bit of Internet of things. Let me just take that back. And then actually showing the part that needs to be fixed and how you unscrew it. So you don't have to be super specialist to be able to change all of these parts or have to wait for an engineer of a particular part to come and... You sometimes call someone out. They say, "Okay, well, I can't change that part. "I don't know what to do," and they bring someone else in. Here, think about where you might need two hands to work but extra layers of information would be useful. And this is my absolute favourite. This is actually what NASA do, meetings on Mars. I absolutely love this because they have loads of 3D data about the surface of Mars and they need to plan where Curiosity is gonna go. So why not actually, let's all look at it together. So this is actually what they do. I think they do have a HoloLens on the International Space Station. That's pretty cool. I'm not sure whether they actually are beaming the guy in. I don't think that volumetric video works just yet, but maybe that's an avatar of the guy. But you can actually have these meetings, not just in the physical space as well, you can share things with people remotely and as remote as the International Space Station. So I've been talking a lot, but let's actually do a demo so you can actually see what I'm going on about. So let me just put the device on. So I'm gonna do some one of those mixed reality capture. This is actually on. Let's switch the device back on. Oki doki. So what you can see there's a slight lag, so apologies for that. And also... Hello? Also what I see is the field of view is a little bit smaller then what you will actually see here. This is like the menu screen. I'm gonna try and not my head around so much because the guys yesterday were getting motion sickness from watching me. So for example, let's add... Oh, before I even do that, let me just show you the gaze. So I'm just sort of moving that little dot around. Don't know whether you can see that. And then when I put my hand... Oop. You should be able to see that turn into a circle, well, a wider circle, and then I can select, I can kind of tap on the world around me. I don't know if you can see that, but it's actually mapping the world around me. Oopsy daisy. One of the reasons why they have like this separate HoloLens processing unit in there is because things change in the world around you. So if somebody walked in now and sat over there, the mesh would be different. Okay. So that's kind of how you might interact with it. But let me place a hologram on this stage. It might take a while to load. Oki doki. So this is quite interesting because they've got a bunch of what's known as volumetric videos. So I think we're gonna go with, 'cause we have a bit of space theme going on, let's have a space man. I'm gonna place him above you guys over here. Maybe that will be good. I can kind of make him a little bit bigger. Say hello to thespaceman. And then there's actually a volumetric video playing. So we should... I don't know if you can really hear him. It might take a little while to load. But he'll sort of like stay here for a while. This is actually the home screen. So what I can do is I can sort of... It's like the desktop of your computer. You can kind of put these applications around the world. So one example here, the Galaxy Explorer. I'm just Let's put Galaxy Explorer. I want to show you this one because, again, bit of a theme of space. So I tapped place that in the world around me. It's now opening up the application. I will turn that down. So what it will do now is you should be able to see multitasking witnesses I need to get a bit better at seeing stuff and talking about it. Okay. The great thing about this is again what I was saying about education purposes and seeing things in a very different way. So when you were studying physics, you probably studied about the solar system. But wouldn't it have been cool to been able to have walk around it? So let me just place that in front of me. You might be able to see the world, and then the solar system. I don't know if it's a... It's not quite as well that, but actually the first time I opened out this application, I put it on my bed and sat on the floor and I was just with it for about 15 minutes because it's just so amazing to actually see the solar system. This is the spatial audio. There you go. Have a little bit of that booming. So let's actually explore the solar system. - [Narrator] Our solar system. This familiar representation of our solar system simplifies sizes at about 4.6 billion years old. The sun is an enormous ball of hot plasma, more than 100 times the size of Earth. About 5 billion years from now, the sun will transform into a red giant and expand to engulf Mercury, Venus, and possibly the Earth. - So if you're interested in trying this demo, do come along and find me later because there's so much more to explore with this one. Realising I'm gonna run out of time, I want to show you one thing that was quite nice that I did yesterday, which, how did it go? Holo Studio. So this one enables you to actually create 3D objects. I did them and then export them. So you might want to create an object. Well, let's just see where it's gone actually, because I think in terms... Oops. Oh no, what's going on there? It's the live demo cast. Here we go. So I have a toolbox here with a whole bunch of things that I can do and there's a whole lot of assets that I can bring into my scene. So I'm gonna create a new digital creation. I don't want to save the other one. I put that in front of me. And then I'm going to maybe bring something into that scene. So how about a set of drums. So you can kind of do this stuff that you would... Well, I've not placed that very well. Let me just move that. It doesn't matter. Anyway, it's spray paint. So you can use voice to choose new things. That's not a very good design. I'm not very good at designing under pressure. However, in proper blue Peter style, I always wanted blue Peter to present this one I've prepared earlier. Especially in.... Don't say that, especially for International Women's Day. Let me just see where it's gone. Ah, here we go. So I made this object. I made the circle bit with the actual logo. I took the logo from the website. I extruded it in Sketch up and then you can export them into here, and then I added some extra bits so I can now customise this. Changed the colour scheme a little bit. Kind of make it purple on this. Okay. You just customise it. And then I can choose to export. So if you've got a 3D printing, you might wanna use this as a way of creating like novel things for 3D printing. So that's just some ideas of some the things you can do. Just realising a little bit late on time. There are a whole lot of other stuff too. Please do come and find me later on. I've got tonnes of other things that, just take it off that, that you can try with the HoloLens. So that's just giving you a taste of what you can actually do with that and what I mean by mixed reality. Okay, so, let me just... So full disclaimer, I don't work for Microsoft. Even though I've been seeing processes of this that are over people that are in the space, they are at the companies in the space, so Magic Leap. I don't know whether you guys have heard of them, but they are a rather elusive startup that have got a massive VC funding based out in Florida. Apparently Beyoncé wasn't very impressed when she tried specialised demo for her last time. But nobody has seen the headsets that they've provided yet. So they're super secretive. They're doing like cover things with optics, but they're definitely the ones to watch. Meta are another company with the headset. They have the Meta one and the Meta two is either out now or not, I'm not sure. But they're in developer version as well. Both of these are in developer version. I think they have like a designer who did some of the design for iMan. So they're really futuristic and not as secretive as Magic Leap. Intel are an interesting one to watch. They mentioned at CES about this thing called Project Alloy. Again, lots of speculation at the moment, about really what that is. I think it's gonna be somewhat closest to a virtual reality device, but with that kind of ability to understand the world around you. So maybe it will do like 3D reconstruction of the world. I don't know. I can talk about like diminished reality and what I think that might be all day. So, again, come and find me later about that. This is not a headset, this is actually a Google Tango device. So there are a couple of fablets and tablets that are already out there which actually have this enabled. What it is, they've created devices with depth sensors on them. So this works, it builds the 3-D map by using the depth sensors to kind of have that understanding of the world around it. If you've got tablets with the ability to do that, then you can actually start doing that spatial mapping too. So this could be quite interesting for volumetric video and they're already available. So you can kind of, as a developer, get hold of one of those. In terms of challenges, is it ready for consumers yet? Well, the biggest issue at the moment is price point, but I'll come onto that in a minute. Field-of-view is another thing. So people generally tend to, when they try it on they go, "Oh, it's actually just a small space that you see," 'cause the field of view actually, with this, is only 40 degrees and obviously we have a much wider field of view. This image in the background is actually from a Microsoft patent that they filed last year. So they're obviously working on how they can expand that. Again, pure speculation. I don't know what the next version's gonna bring. But that's something that I think they will want to improve upon. Although having said that, having use this a lot, there's kind of an affordance that I have for it. I kind of get used to the fact that it's a small space. So I don't know. I personally feel that you could, at the moment, put this out to consumers, but the biggest problem is the price. So you're looking at 2700 pounds for a device like this at the moment. So, realistically, It's not gonna come in the market for consumers for a few years, I reckon. I think you're looking at 2019, 2020, when something like this might come available because the guys over in Microsoft, they've been quite vocal about the fact that they would like to get it underneath more into the $1000 realm then the $3000 realm, but they don't want to compromise on the kind of experience that you have. So what they're actually doing is they're bringing together some other immersive headsets which are far more cheaper and they use similar technology to this, much closer to kind of a virtual reality experience and they're kind of billing themselves as Microsoft mixed reality now. So you can buy a cheaper headset for less than 300 pounds is possibly what they're looking out for these other devices where you can just plug it directly into a Windows machine and have an experience. So what you can start doing is if you're building for that, you can potentially get hold of one of these like later in the year. But this is the kind of thing that you would see with that type of headset where it's more virtual, but you are aware of the surroundings. So there's actual kind of mesh around you as your play space. So you're not so tethered as you would be or not as limited as you are with the current Oculus Rift and HTC Vive. So if you want to build for mixed reality, yeah, you can buy one of these. They're a bit of an investment. To be honest with you, like you can have one in a company and then create the meshes, capture the meshes of the rooms around you, and then give them to other members of your team who can build with the emulator. You can buy these sort of cheaper headsets as well. This is the link you need to go to. All the info is there. Also that Galaxy Explorer that you saw, all of the code for that is on GitHub as well, so you can see how to build. You can get started today as well. You don't need a headset. If you've got access to a laptop that will run Windows, you just need Visual Studio, Windows 10 Pro edition if you want to run the emulator. Unfortunately Windows 10 Home doesn't quite do the virtual machine to load the emulator. So you will need to upgrade. So if you could actually get a student licence, but it's about 120 pounds to upgrade to Windows 10 Pro. And then if you've ever created a universal Windows platform app, like anything that has been built for universal Windows platform, you can actually run in this, but it might be more of like a two-dimensional screen. But that's the easiest way you can get started is just converting an existing app that you might already have like I had Candy Crush on my wall the other day that I was playing. It was quite cool. But if you want to actually do a much more immersive experience, you probably want to look into using unity game engine because that gives you... Unity's been around for a very long time. It's a lot of game developers are experienced with and there's tonnes of tutorials, tonnes of support around the community for that. In terms of the community, a couple of useful links just before I wrap up. So there's actually a couple of Meetup groups you can join here in London. So the HoloLens user group will be having its second ever meeting tomorrow, I think. So come along to that. It's a small group at the moment, but hopefully it will grow and if you've been inspired, you just come along because you can try on headset and talk about Unity and it's really great. And then Mixed Reality London, I'm organising this. We're looking to kind of get started around May time to have our first talk. And also, the Immerse Network as well. If you're working in the virtual reality, augmented reality, mixed reality spaces, look them up because that's partly funded by... The government funds Innovate UK. Innovate UK have a whole load of initiatives and one of them is this knowledge transfer network. And as part of that, they have money for this Immersive Network. And what they do, they've got loads of information about all the events that are going on and also letting you know about funding opportunities as well because, let's face it, a lot of this is research and development, right? And so, if you can kind of create something and get some R&D funding from the government via Innovate UK, that's pretty cool. So check out those. I would just like to thank you all for your attention. Yeah, I'm gonna be around all day. Come and try out the HoloLens downstairs and I'd like to open to any questions.