!= Introduction to Three.js - 3D Modelling in Javascript - Sessions by Pusher

Introduction to Three.js - 3D Modelling in Javascript

Mairead Buchan speaking at Bristol JS in February, 2017
406Views
 
Great talks, fired to your inbox 👌
No junk, no spam, just great talks. Unsubscribe any time.

About this talk

A run through of the basics of Three.js and how to create a rigged hand model in JSON.


Transcript


- [Mairead] Hi everyone, I'm going to talk to you about Three.js. So this is a really basic intro, it's not much more than a hello world. So if you've covered this before, there might not be very much new here for you, but if you haven't seen any Three.js, then I think it's really exciting what you can do with JavaScript these days. And I just want to show you a little bit of work that I've been doing based on this. And I've been a JavaScript developer for about 15 years, which is a bit of a dinosaur. I specialize in JavaScripts from front-end because I really like seeing interaction happen, that's the exciting bit of web development for me. And plugging other things into web browsers as well and seeing what you can make happen. And I kind of started off with this 3D stuff using a Connect and a bit of Leap as well, I'm really interested in the interaction. I'm going to apologize now for the codes here, there's quite a lot code slides. So when we're talking about doing 3D stuff in a browser, and what we're using is GLSL. We're using WebGL, which is based on GLSL, it's based on an open standards called OpenGL, and the SL is a shader language. So what it is is these components called shaders, they come in two flavors, a vertex which is kind of the angles of a shape, and a fragment which is the surface of a shape, and that's what makes all kind of 3D objects. And GLSL is written a bit like C, and it's kind of specifically written to communicate directly with your graphics card. So it's really optimized for memory performance, so obviously there's a lot of processing going on and rendering animations of 3D, so it needs to be quite low level. And you can write your own shaders, if you want to, in this syntax, and then attach them to JavaScript in a string. So you can write that if you want to, but it's not trivial, I wouldn't necessarily recommend it. And WebGL is kind of built on top of that, so what WebGL is, is sucking all of that shader stuff into JavaScript, and then also exposing you a browser JavaScript API like all of the other JavaScript APIs that let you communicate with that kind of shader level, and with the graphics card. And so you can write all of that stuff yourself if you want to, but like a lot of JavaScript APIs, particularly browser, there's differences in browsers, it's really gross doing that kind of development. So what you really want is a library that sits in between you and all of the browsers, and that's where Three.js comes in. I would say that Three.js is probably the best thing to be using if you want to start with this, it's the most popular and it's got the most community around it. And the guy that wrote it is this guy called Mr. Doob, he's been around longer than I have, he used to make really cool flash animations, he made really clever mass particle, like, wave animations and really clever stuff, and then he kind of disappeared and I hadn't heard anything about him until I started learning about Three.js. And then I suddenly realized it was this guy, Mr. Doob. I think he's Spanish, I think his name is Ricardo really, but Mr. Doob is a cool guy. So yeah, this is an abstraction layer basically over the top of that WebGL communication in the shader language. What's really amazing about this library and why I think Mr. Doob is such a genius is that he's exposed a really simple set of instructions and API over something that's actually massively complex, like doing all of your...like the maths of geometry. I tried to get into it and tried to teach myself stuff, unless you did higher maths as an academic subject, it's actually really complex to get your head around. But Three.js makes it really simple, and it's very broad and you can get a lot straight out the gate. So it's really fun to work with. It's based on a couple of major components, or kind of the structure of the library. And so what you've got is a camera that looks at a stage, so everything you're making, like a theater stage or a movie set, you put everything on a stage and then a camera points at it, and that's what's being rendered out to the canvas context. I think I skipped that earlier, it's canvas, like a 2D canvas object but it's using the WebGL context of a canvas, if you use canvas and you set the context to 2D, this is 3D. So you've got your camera and it's looking at the stage, and then you can put things on the stage, and Three.js gives you loads of geometry, core components, and things that you can add to the stage. And then you've got these two other things, lighting and materials, and they go hand in hand, you can't have a material unless you have a light hitting it and bouncing off of it. So you can't get a 3D object, it won't look three dimensional unless it's got shade and shadow or it throws a shadow. So lighting is actually a really important concept to Three.js and 3D modeling in general. And the way that light hits the material and bounces off it is going to describe how that material looks to your eyes, so a matte material reflects light differently to shiny materials, so that's quite an important...kind of they go hand in hand. And so now I'm going to kind of leap into a bit of codes, I'm not going to read through it and I'm not going to insult your intelligence by live coding in front of you, because that's really painful I think. But I'm just going to show you some examples of code and then show you what it looks like on the screen. So I've knocked up a bit of Boilerplate, if you want to mess around with this at home, that sets out a stage already so that you can play around with. And I'm going to show you what that looks like in code. So not that one, that's later. So we start off here. Can everyone read this? You all right at the back? So we start off by adding a viewport object, so it's just a bit of HTML so that you can shove something in it, and it needs of width and height. And then we set up a renderer, the renderer is the thing that packages all up the Three.js and spits it out on to the canvas object. And then we create a scene, so this is like our movie scene, our theater stage. We need a camera that points at the scene, we're kind of adding the camera here, but you can set up its angle, and its view, and whatever. Then we set up some lighting, this is just like a kind of a global light, it's going to be like a sunlight lighting the scene. I've got a little bit of a helper here, this GridHelper is really useful because it shows you your perspective, so when you're looking at the stage you can see that you're looking at a 3D thing, before you put any lighting on it you can see perspective. So he's got a couple of helpers and lines and stuff that you can put on to see it. And then the last thing we do at the bottom is we tell the renderer to render, and that's quite important, that has to come at the end when you're setting it all up. So that's basically all you need to have a Three.js scene. And then you can add objects to it. So I'm starting off at the moment with a basic material. To put an object on the screen you have to have a mesh on the material, otherwise it's not a solid thing. And so we're just using the basic, basic doesn't reflect any light. And I've just create a sphere here, so like all of the objects have their own premises, I think that's radius or diameter and size. And then you position it, and these are obviously like X, Y, Z coordinates, so that's quite important as well, you're dealing with Cartesian coordinates when you're working in three dimensions. And then you just add it to the scene, and then you've got a little mesh on the scene, hang on a minute. So that's what we've got from that little geometry. So can you see here that that doesn't look like a 3D object? So that's the basic material, it doesn't reflect any light. So you can't see that that is a 3D thing, is a 3D thing in Three.js, but it doesn't have a material that shows you that that's what's happening. So then if we add some materials, you can add a couple, this Phong and Lamber I think are named after the people that made it, they're quite similar, that's what you get out of the box in Three.js. You get two or three materials already, and I'll show you normal in a bit. So this, I think, is Phong. So can you see that? That's a 3D thing now, you can see it, and all that really is is that it's reflecting the light that was hitting it, but because it didn't have a material before you couldn't see that that was a 3D object. And then if we change Phong, this is quite subtle, so you might not see it change. It's changed size, that's not particularly helpful. I don't know if you can see, like the Phong is slightly shinier, so the light is hitting it slightly differently. I don't know if you can see it better on this screen. Not that one, sorry. I said I wasn't going to do this, didn't I? There, anyway, let's get to the next one. This is a spotlight, so this is kind of illustrating what I'm talking about again in a different way, this is actually a spotlight hitting the object, it'll change the color of it. This is still a Phong material, but because I've changed the lighting it makes the ball look shinier because you're hitting it with a more intense lighting. So lighting materials go hand in hand and they do, they react with each other, so when you change the lighting it will change the nature of the material that it's pointing at. So here, I'm just kind of... I've added a spotlight, so it's still got a global light hitting it. I'm getting too close to my microphone. It's still got a global light hitting it, but then I've added another spotlight to the stage, and that has changed the aspect of it. And then, this is quite important as well, when you change something on the stage, you need to update the material because it needs to re-render itself, and you need to change, just call the render and tell it that it needs to change something again. And then you can change the camera as well. So if you can imagine, if you are animating a complex scene you can obviously move all the objects in a scene using JavaScript, you just change the X, Y, Z positions. But you can also add cameras to scene, and you can rotate them, and you can point them at objects. So you can either look at the scene or you can look at an object, so you can animate a camera around an object or you can animate your camera around the scene. So you can do those kind of cool matrix effects, where they kind of pan all the way around, 360 around an object. So you can do all of that kind of stuff. I'm not going to show you that demo because it's quite boring. And the last one is animating a scene. So using requestAnimationFrame, if you haven't done any animation, it's really important to do that instead of just adding a timer loop or anything like that, because requestAnimationFrame, if it's struggling to render, and particularly 3D stuff is quite processor heavy, if it's struggling to render, it will throw away frames so that it keeps up and your animation doesn't lag. So if you're calling any kind of animation, you've got to use requestAnimationFrame. And then you just call the render function, so here, all I've done is I've told the camera to change position, and then it's incrementing in a render loop and then it's calling the render. And that... I just closed my presentation. All it's doing is animating the scene, but it looks like the object is moving, what you're actually doing is moving the camera further away from the object. And that's it really, that's the basics of all of Three.js in a nutshell. I've just closed my presentation. I think I'm here. All right, where did we get to, moving a camera, animating. So that's pretty much it, that's the hello world of Three.js, it doesn't really take a lot to get that far, that's less than an hour's work really to get all of that together. But that's not really going to give you very much, if you want to do proper 3D stuff, presumably you want interesting objects and not just spheres, and torus, and all the rest of it, you want to create a 3D model, so if you're making models, you really want to be using 3D software, and then you can import that into JavaScript. So there's a couple of main competitors out there, 3D Studio Max and Maya are what professionals use, they're very expensive, and they're not for an amateur web developer to just pick up. But Blender is the free kind of community version, it's actually really decent, you can do pretty much everything in Blender that you could do in 3D Studio. And it's got a huge community, loads of tutorials, loads of help pages, people answering questions. You can buy stuff as well, if you don't want to spend the time making it, you can just buy an object like Clipart and do your own stuff there. And then what you do is you export it out to JSON. So I'm going to show you a little bit that I exported, and kind of run through what you get in the JSON file. So you can see here, like you've got the vertices and the faces, normal is, kind of, it's the perpendicular angle off of a face, UVs and materials, so it's got lots of kind of metadata, and then it's got loads of other kind of...that's the material data for the material that I'm using for this hand model, and then loads of other vertex data for a complex object. And then at the bottom, what I actually want to show you is this, you can get other objects into your... so this is like a JSON file that's going to get imported into Three.js, but you can also add a lot of these other properties for the purposes of animation. So I'm actually using a LeapJS device, and the points of a hand to map X, Y, Z points, and that's exported alongside in that. So what I did is I bought a hand model from Blender, and then I added my own bone structure in, which is something the Blender can do, it's called rigging. So it's like a skeleton and it will do all the kind of inverse kinematics of a hand, is a realistic shape. And then it will export all of that data as well, and it exports it with a name, so that you can then address that in your JavaScript animation, so when you're moving stuff around screen you can position it. So yeah, it's got the data for the vertex and the fragment, it's sort of saying that's what makes your mesh objects, and then it skins it with a material and it contains the material data, and then it's got all the other properties you need for communicating with that and controlling it, and then you import it using a Three.js object. So what I was making, it was actually part of a VR project, I was using a Google Cardboard, and trying to get the data out of a LeapJS device, through the computer, port it over WebSockets, so I could use a mobile phone inside of a Google Cardboard. I've got to say, having done that experiment, I wouldn't recommend it, it is doable but there's so many loops to jump through, that if you really wanted to get something working, as like a WebVR application, you'd be loads better using the WebVR API, and using like an Oculus Rift or something that is designed for that purpose. Like, the Google Cardboard is really good if you don't need it to communicate with anything else, but having to port data over WebSockets to the phone and then also, if you're using the browser to do this, you have to tell the browser not to turn itself off because your phone goes to sleep if you're not using your phone. And also the Google Cardboard has a magnet on the side of it, which like corrupts your screen, it's not a good idea. So if you want to do WebVR stuff, if you're interested in Three.js and that why you've come and you want to do WebVR stuff, stick to the WebVR API and stick to a Rift or the competitors. It was interesting, I'm not saying don't do it, maybe you're curious. And my code is on GitHub, so if you want to see what I've made, then you're welcome to go and have a look and see what I did. But what we wanted was a moving hand, so we had a LeapJS and we really wanted to have a hand moving in front of us. Part of the application was that we wanted someone to think that when they were looking out they were looking at their own hand, that was quite an important part of our application. So that's why I was using Three.js to rig this up. And in the end, Leap wrote their own hand plugin, which was loads better than the code I had written. So I came back to this project six months later and discovered they'd already done it, but I said, "Throw it all away." But it's good because theirs is better than mine would've been. And so what it's doing is it's picking the points of a hand, and the bone points, that's how Leap works. And then I've hooked it up to a skeleton I've created that maps to the LeapJS hand points. And then also the LeapJS had it's own kind of frame animation that it's using. So, let me just reload this page. Can you see there? This is just an example of what you can do with Three.js. And it's really not that complex either because this is using all of the same technologies that I've just showed you, so a hand model that's got some rigging, that's been written in Blender or similar, and then mapped to a LeapJS. That's only a couple of weeks work, but it's actually really successful and you can put it in loads of web games and loads of other stuff, if you're interested in that kind of thing. So that's it really, that's everything I have to say. I do have a couple of plugs however, now that I'm stood up in front of you. I teach a bit of JavaScript as well, and a bit of front-end. I'm currently working with an apprenticeship scheme, if you know particularly any young people or people that you think would really...that aren't interested in university and really want to go into the career that you do and you love, because you wouldn't be here unless you really loved what you were doing, if you know any young people, we are rolling this out into Bristol in the coming years. So they're working in Leeds or Manchester at the moment. It's partly government funded, and it's partly funded by the company who is going to take them on. So also if you are a company who happen to want to hire, and you'd really like to hire juniors and you'd be interested in placing apprentices, then please get in touch, either go to their web page, they've got a sign up form to say that you're interested, or drop me a line on Twitter and I'll record your interest. And I also teach on another residential training course, they're also rolling out online tuition. So if you know anybody who is really jealous of you because you've got a cool job and you really like it and they really want to be like you, or someone... These guys are mostly older people, they either come out of university and they're looking for a career, or people who are looking to change career, if you know anyone like that, these guys are really great, and they teach Ruby Stack, and they teach you to be full stack, and then they also try and get you a placement as well. So they're kind of like, they take you straight through from being a talented beginner to being a fully qualified web developer who can work. So that's it, I've got a...you can reach me on Twitter, you can look at my code on GitHub under Mairead. You can look at my slides here on my website, lcw-workshop. You can also look at my Cardboard stuff, on GitHub, if you want. That's it, has anyone got any questions? - [Male 1] Yes, I do. Does Three.js have any spot for Physics, do you know? - Yes, yes. We didn't get that far, that's what we wanted to do, so like once we'd got a hand model working, then we wanted to get some collision detection running because what we wanted to do is be able to reach out and touch something, and that was going to be our scoring mechanism. So, yes you can. And they tie in really nicely together, because basically you've got a mesh in Three.js, and that maps straight to the mesh in Physics, so all you really need to do is make sure that those two objects know that they're the same thing and then you've got your collision detection. So it looks fairly straightforward. - [Male 2] [inaudible] more questions. - There's one here. - [Male 3] So my question is, in your hand model, it's actually got lots of different parts to it that are joined together and they rotate around the same knuckle joint, and that's all done with Blender model, you haven't had to code any of that? - Yeah, pretty much. You can lock in kinematics so that you can't like bend a finger back, so you can program all of that stuff in in Blender. Yes, it's all part of the rigging. So you can control the rigging and say whether or not it goes in certain angles. And I believe, and the reason that the LeapJS hand plugin is better than the one I made is that my hand was bending back in weird ways, and because they've obviously got a whole team of developers making something better, they put a lot more thought into making that like a natural hand movement that can't do strange things. And they've also put a lot more work into like occlusions, so if you hold your hand up it knows that there's still five fingers there, whereas when it first came out a couple of years ago, if you turned your hand straight on, all the fingers disappeared. So they've put a lot of work into making that a useful thing that you could use like in web gaming or anything else really. - You have one in the back. - Oh, there's one right at the back here, I don't want to try throwing it. Yeah, and so the questions was, what's LeapJS? LeapJS is a JavaScript API that they've built for the Leap Motions, they do have other APIs and you can do it in Android and other languages, but because I'm a web developer and JavaScript developer, I'm using LeapJS. It's actually pretty decent, there's not a lot to it really, it's pretty small because all you're really doing is controlling the hands. It's got some gestures, so if you wanted to do gesture control, but I'm not convinced that that's a particularly useful way to use a LeapJS because I don't think people do want a gesture. Apart from kind of kiosk stuff, I don't think people do want to make hand gestures at their websites, but in terms of doing VR, and 3D animation and gaming, particular web gaming, and like the things that I'm really interested in, I was working in the health industry, I was trying to make a bit of rehabilitative software that maps an actual human hand. So that's where I think LeapJS is going to be more useful, and I think that things like WebVR are going to be more useful. So people think, "Oh, it's never going to catch on," and it's only for gamers, that's because it's only just hit the market, and like in another 10 years it will be everywhere. So, and I do firmly believe that whether or not it's in the browser, or not, but I do think that we've only just touched the start of it. But I like JavaScript, why not do everything in JavaScript, if you can? If I'm from preaching to the choir here, you know? Anymore, anymore? - [Male 4] Slightly off topic, what do you use Visual Studio for? - That's my new client, that's my new client. Data visualization using a .NET MVC and Metronic, which is a template that uses AM charts to do visualization. It's fun, I wouldn't... the Visual Studio is horrible. I've been using a Mac for years so I've just stepped into using a Windows laptop, and now I don't know any of the shortcuts and I don't... click CTRL, CMD, CTRL. Yeah, I wouldn't recommend that either, to be honest. Like, NEWCARE is their package manager, it's not MPM, it's not. Anymore, I don't want any windows fans [inaudible]. Anymore? - Can I have another one? - Yeah, yeah. - So you took [inaudible] in Blender, you had a very simple 3D object type of box, and you use Blender, can you create your own shapes from primitives, just from vertices? - Yeah, yeah, you totally can. You can make a mesh and then you can join them together, and then Three.js has commands that...like you can in Photoshop, or Illustrator merge objects together, and you can dig stuff out as well as you can exclude objects. It's just that it's very time consuming. And like, if you are going to do anything really complicated you'd be better off using Blender than you would doing it in code. But this is one of the things that I think 3D gesture modeling would be loads more useful, because when you're making models in 3D software, you have a two dimensional interface to a three dimensional object, and it's really painful because you have to do a lot of switching contexts, and like a right click does something different, because right click lets you drag and then you've got loads of right click, left click menus to give you that kind of ability to spin it around and do stuff with it, it's really time consuming. So there has to be a way that you could just control that object with your hand and spin it with your hand and change it. And I've seen the other stuff that people have done with Three.js, like mapping a sphere, and then they had four projectors and four mirrors so that they were projecting a hologram, and then they were using a LeapJS to spin the sphere like in Iron Man, and they had like a rotating hologram in the air. Like, that was just built out of web software, that wasn't built out of anything really complicated. Yeah, it's really amazing. So, and that's not, we're not really that far off of that kind of crazy Iron Man stuff, because if you can make it here, like a guy on MIT for his PhD project made a rotating hologram then, you know, that's... - [Male 5] If he can do it, anyone can. - Anyone can do it. But what I mean is, like, that's, that was two or three years ago. So that was his pet project two or three years ago, so if that's what could be done two or three years ago, then what could be done in the next coming years? Everybody could get their hands on that kind of stuff, and you could do that tomorrow, don't sell yourself short. So yeah, yeah. It's good stuff. - [Male 6] Any really sick [inaudible]? - Do you mean that spinning sphere? - Yeah, possibly, or what's [inaudible]. - I can probably find it for you. I think this was his name. Oh, I could see that full screen. So that's all his Three.js animation that he's using a LeapJS to... Is this though his hologram? Oh, that's just his control, there is a hologram as well. Sorry, I have to dig out to... It is in older slides of mine. I've demoed that before in front of people, that he's got. So that thing that you saw him controlling, he's got that. I better turn this off in case something unsanitary comes up. Oh, oh, yeah, yeah, okay, here we go, yeah, so. So he's got like four screens that he's mirroring up. - [Female] I can totally build that [inaudible]. - And so that video that you just saw of him rotating the sphere, that's what he's doing now. - [Male 7] [inaudible]. - [Male 8] You don't have that in your garage? - It's cool though, isn't it? Yeah. And that's just using Three.js and then mapping a photographic image onto a sphere, so using that kind of projection mapping to put a picture on it. - [Male 9] Please, a huge thank you to Mairead.