How you can become an AR/VR developer

A guide for developers that want to become AR/VR developers and find related resources.

Introduction

This blog post was written under the Pusher Guest Writer program.

The Augmented Reality (AR) and Virtual Reality (VR) industries are still in its infancy, however, this is why now is the perfect time to learn how to create content for these technologies. They need it to make the jump to the next level of adoption.
Luckily, from a developer standpoint, the skills needed to get into AR and VR are very similar and the barriers to entry are still relatively low. This article will give you an overview of how to get started.

The Most Important Skill You Need

No matter whether you’re a newbie who is starting with programming or a veteran with years of experience in the field, the main skills you’ll need to become an AR/VR developer are in the 3D area. In fact, it’s not very different from 3D games development.
Why? Because AR and VR are about creating immersive worlds or environments that can be interacted with in three dimensions, like in real life.
So depending on how deep you want to go, you may have to learn about 3D modeling and/or scanning, 3D games engines, 360° photos and videos, maybe a little bit of math and geometry, programming languages like C/C++/C# and software development kits (SDKs), and how to design experiences for users in 3D.
So let’s take a deeper look at each of the topics you need to address to become an AR/VR developer, focusing first on VR.

Hardware

Augmented/Virtual reality (and 3D development in general) demands high-end hardware. However, as the price of hardware is constantly going down, this may not represent a big problem as it did a few years ago.
If we take a look at the requirements of the two most popular VR platforms (HTC Vive and Oculus Rift), we’ll see they are basically the same. At the time of this writing, these are the recommended hardware specifications:

Processor: Intel Core i5-4590 or AMD FX 8350
Graphics: NVIDIA GeForce GTX 1060 or AMD Radeon RX 480
Memory: 8 GB RAM
Ports: 3x USB 3.0
Operating System: Windows 7 or superior

Does it mean that if you have an i3 processor or a GTX 1050 card you won’t be able to do any development work?
It depends.
The above are the recommended specifications to have a smooth experience, so for some projects, maybe a little less would do the work, and for others, maybe you’ll have to compensate for the lack of power in one area with an increase of power in another one (for example, by pairing an i3 processor with a GTX 1070).
There are minimum requirements and tools to make sure your hardware is compatible (here and here), but the suggestion is to try to aim for the higher end and use desktop PCs to have an easy (and cheaper) upgrade path.

Virtual Reality Devices

In this area, we have many options. Let’s categorize the most popular by their degree of freedom (DOF), which refers to the ways in which an object can move. We have two options: three and six DOF.
Three DOF means that you will be able to interact with the virtual world in three dimensions (in an X, Y, Z coordinate system) by moving your head using a head-mounted display (HMD), however, you wouldn’t be able to move forward or backward.
With six DOF, you can move forward/back, up/down, and left/right, so you now have three more types of movements, hence the name. You can learn more about degrees of freedom in this article.
The following devices support three DOF:

While the following can support six DOF:

Of course, each device uses different SDKs, programming languages, and has different constraints, but you’ll find that they have some things in common:

The principles for designing a Virtual Reality experience are the same
Most of them are compatible with motion controllers to interact with the virtual world
Three DOF devices use smartphones as head-mounted displays
Six DOF devices use desktop headsets

At this point, and depending on what you want to accomplish, you may want to do some research and choose one or more device(s) to develop on. Of course, you can do more things with desktop headsets, but you may want to start with something simple, and transfer your skills from one device to another later. It’s a matter of learning the right tools for the job.
Now let’s talk about the most commonly used software to develop VR content.

3D Game Engines and Programming Languages

A survey published in The State of the Developer Nation Q1 2017 report puts C# and C/C++ as the most used programming languages for AR/VR development. This is not a coincidence. The most popular game engines you’ll need to learn use them:

Unity, which uses C# as its primary programming language.
Unreal Engine, which uses C++ and a node-based language called Blueprints Visual Scripting.

The good news is that all VR devices have SDKs available for both engines so you can use only one of them to develop AR/VR applications and target more than one device.
The bad news is that the learning curve is relatively steep for both. When in doubt, most people recommend Unity because it’s easier to learn and more resources are available. However, Unreal can offer you better graphics and more power.
Another thing to take into account is that Unity has many versions with different costs, but there’s a personal version that is free to use. Unreal is free to use, with a 5% royalty on gross product revenue after the first $3,000 USD per game per calendar quarter from commercial products.
The recommendation is to try both of these engines to see which one suits you best.
In addition, it’s worth mentioning that Google provides SDKs for Android (in Java) and iOS (in Objective-C) to develop for Daydream and Cardboard devices.

Make or Find Assets?

The first things you’ll need for AR/VR development are art assets, especially 3D models. You have two options here: make them yourself or use models made by someone else.
Making 3D models by yourself is the most difficult option, but in the long run it may be the best (and most cost-effective). If you choose this path, you’ll have to learn to use programs like:

A technology that can help you create your own models is 3D scanning. Basically, things captured by a 3D scanner in the real world become a virtual 3D model. They may not be perfect yet, but they can help you get started, and there a lot of options with a wide range of price points. Some of these are:

Otherwise, you’ll want to get 3D models from places like:

This option is fine when you’re starting and you don’t have the skills, but sooner or later, you’ll need to make modifications to get exactly what you want, so the suggestion is to combine the two options while you learn to make your own models.
Something similar happens with 360° videos and photos, used in Virtual Reality to provide an immersive experience.
You can buy a 360 camera like the RICOH THETA or the Samsung Gear 360 (only compatible with Samsung devices) to create your own media or use sites like Flickr or other sites for royalty-free stock images and videos.
In the area of audio, we have it easier because there are no special requirements; we can use the same sound effects used in traditional games and videos from royalty-free audio sites like FreeSound, and audio recording and editing software like Audacity or Adobe Audition.

From Web Development to Virtual Reality

Maybe you have a web development background and you’re wondering if your knowledge of JavaScript can be of any help.
Luckily, WebVR is an open standard with a JavaScript API that makes it possible to experience VR in your browser.
It works with most VR devices, but you don’t need any of these or plugins as any compatible browser (even in your Android/iOS phone) will do the job. You can check for the latest browser support at WebVR Rocks and find more information on the Mozilla VR site.
Although knowledge of 3D terminology and modeling still applies, if you know JavaScript, it can be a good choice to start with WebVR instead of a VR SDK, Unity, or Unreal.
In this area, there are two frameworks worth mentioning:

A-frame. A framework for building virtual reality experiences with HTML and an Entity-Component-System approach. It was developed by the Mozilla VR team and provides one of the most powerful ways to develop WebVR content.
React VR. A new library developed by Facebook based on React and React Native. It allows you build VR websites and interactive 360 experiences using the same declarative component approach than React.

As both are based on Three.js, the most popular Javascript library for using 3D graphics in a web browser, knowledge of this library will give you a strong foundation.
Unity has the option of writing scripts in a language similar to JavaScript called UnityScript, but it’s not the same as the JavaScript you’ll use to develop for the web, and C# is the recommended language for Unity.

And what about AR?

We have been focusing on Virtual Reality, but Augmented Reality is a different concept.
Virtual Reality is about creating a virtual and immersive environment to replace the real world. Augmented Reality adds layers of data to the real world instead of replacing it. For example, it can translate the words you see in another language, or it can make you see monsters when you walk near some place. In contrast, Virtual Reality can take you to another country or a land full of monsters.
When you join both concepts, you get what is called mixed reality, where you interact with both the physical and virtual environment.
However, from a development standpoint, VR and AR are pretty similar. In fact, you can use Unity and Unreal (with the help of some plugins) to develop AR content. For example, a simple AR app will recognize an object and present a 3D model that you could manipulate as if it were real, so the skills needed for VR are applicable to AR also.
One of the most popular tools for developing AR is Vuforia, which is available for Unity, Android, and iOS, provides a lot of features, and support for many devices, including Tango phones.
Talking about devices, Augmented Reality can be used with head-mounted displays, but also with other devices like smartphones, glasses, and specially-made visors, perhaps being the most popular Microsoft HoloLens (although more inclined towards mixed reality).

User interface (UI)/User experience (UX)

The AR/VR industry is new and therefore there are not many best practices yet for developing this kind of experiences, but we should take for granted that they are different from traditional 2D apps. For example, how do you handle input? A keyboard in a virtual world may not be the best choice in some situations.
On the other hand, one of the biggest problems about VR is simulator/motion sickness. People can get sick either by lagging, unnatural movements, and mismatches between physical and visual motion cues, among others.
Here are some resources on the topic:

In summary, what are the steps to become an AR/VR developer?

Define your platform. Decide which devices to target, which platforms (mobile, desktop, web), and which game engine/SDK/framework to use.
Learn the skills. Learn about the terminology, 3D modeling, the language of that engine/SDK/framework, UI/UX for AR/VR.
Implement something small. Although a great number of AR/VR apps are games, there are a lot of areas that can be targeted, like education, data visualization, 360° experiences. The key here is to practice and gain experience.
Join and engage in VR communities. Go to meetups, participate in forums/chats, talk to other developers, ask to the community.

Resources

Pusher Tutorials
– Building a realtime React VR app

Free Courses:

Paid Courses:

Become a VR Developer – Nanodegree

Blogs:

Podcasts:

Sessions by Pusher videos:

YouTube Videos:

YouTube Channels

Slack Channels:

Who you should follow?

Tony Parisi | Co-creator of VRML, VP @ WeVR
Benjamin Lang | Co-founder & Executive Editor of RoadtoVR
Bruce Wooden | Co-Founder AltspaceVR
Kent Bye | Voices of VR
Sanem Avcil | VR Influencer
Rob Crasco | VR Influencer
Malia Probst | Host of the Real Virtual Show
Rick King | VR Influencer
Sam Watts | Director of Immersive Tech make[REAL]
Eric Chevalier | Co-founder of VRScout
Liv Erickson | AR/VR developer, author, evangelist
Helen Situ | Founder and editor of Virtual Reality Pop
Eva Hoerth | VR Influencer
Kevin Ngo | Mozilla VR Developer
Amber Roy | React VR Developer
Andrew Imm | React VR Developer

Conclusion

The technology is not quite there yet. It has a lot of limitations, it’s expensive, and standards and best practices are still emerging. However, it has great potential, and now is a great time to get started – especially considering that most professional AR/VR developers have less than two years of experience.
Besides, some of the skills you’ll learn (like C#) can be transferred into other areas of software development or game development if you end up deciding AR/VR is not something for you, making it a low-risk/high-reward opportunity.
Do you have anything to share? Let us know in the comments.

Docs

Tutorials

Support

Glossary