VR, AR & spatial

Cross-device VR and AR — input systems, spatial audio, comfort, and keeping up with a decade of changing headsets.

Technologies: WebXR, WebVR, Vuforia, 8th Wall

I started making VR-ready platforms shortly after the Oculus Rift DK1 was released in 2013. Since then I've worked directly with most of the headsets: Rift, Gear VR, Go, every Quest version, HTC Vive, PS VR, Daydream, Windows MR, Pico, HoloLens and Apple Vision Pro. I've made dozens of VR and AR experiences, built UI systems and improved upon WebVR polyfills and Three.js VR support.

Binaural Processing with Motion Tracking (2013)

This was the first time I experimented with 6-axis tracking and virtual space. On this real-time 3D binaural simulator, I used sensors to track the user's head orientation and anchor the source of sound relative to it in a virtual space.

I also did real-time HRTF filtering, which reproduces how the sound transforms as it reaches our ears and ear canals from one specific direction, allowing our brains to identify the emitting source without seeing it.

It turned out to be a very valuable field of research as Oculus DK1 was about to be released, triggering a new wave of VR.

Virtual Home Visit (2014)

This was the first serious game where I implemented VR support. At the time, Three.js relied on an early specification of WebVR (later WebXR). I had to use WebVR polyfills as VR support wasn't implemented in all browsers. Back then users had to turn on an experimental flag in Chrome settings to enable VR on the web.

For this version, I had to manually adjust the rendering pipeline and tweak many values so the environment was correctly perceived while using the headset. I also had to design against motion sickness by teleporting instead of animating the camera, and adding fade-to-black effects to smooth transitions.

Hyperspace (2014)

When the authoring platform started growing, we extended VR support further. VR had been in the plan from the beginning, so it was considered whenever a feature was designed and implemented.

I created a custom input system where authors were able to navigate, interact and edit scenes with any available input (mouse, touchscreen, VR controller or gaze).

I added AR support through the 8th Wall library, which allowed us to place avatars around the user in the real world. I also created experiences where complex machinery could be inspected by physically moving around it.

I added features like VR notes, whiteboards and the ability to pick up items, and experimented with positional audio. I also created audio zones, where groups of users can get together and have a private conversation in a big hall. Audio zones can also be locked so only people with a code can join.

As headset iterations came to market and were replaced by new versions, I always made sure to make every feature compatible with the device. Some of them were 3DoF (degrees of freedom), others 6DoF. Each one reported height and handled the initial position differently. Their controllers were mapped differently. Browsers changed their specs and APIs over time too. VR was evolving fast and we had to keep up.

One of the hardest challenges was supporting audio input and output in some of the headsets. Audio calls and speech recognition were critical to many clients and some headsets had permission and encoding limitations. Doing regular testing with real devices was essential to maintain good VR support across the platform.

Over the years I've designed many VR experiences. I've learned all the particularities of the medium: how to design comfortable environments, how to build interactive and engaging mechanics, and how to communicate with the user effectively without being intrusive or overwhelming.

Speak It Yourself VR (2019)

It didn't have any 3D models in it, but it was a very interesting project, as it taught me a lot about 3D environments.

Speak It Yourself uses 360 video footage of people talking to help you improve your language skills. I learned a lot about 360 video rendering, the different video formats and their pros and cons, compression and shaders.

I created a VR HUD where the user can pick their answer. The UI follows the user's gaze and can be interacted with using hand controllers, or with gaze alone when no controllers are available. I learned where to position the HUD so it stays comfortable and unobtrusive, and how to move it as the user turns their head without inducing motion sickness. I also learned the importance of visual cues in VR experiences.

HistoPad (2013)

HistoPad ran on a tablet that was handed to visitors in heritage sites. We had 3D reconstructions of the sites and the goal was to use AR to show how the architecture looked in the past.

After many attempts (GPS + orientation sensors, AR markers), we settled on using Vuforia in Unity to align the 3D space to the real world around the user.

I used the plugin to align the 3D model to the user's view, added virtual markers with information about points of interest, and created a quest system: a list of items to collect, each unlocking a piece of knowledge when it was found.

Where I've used it

View as Markdown