Binaural Processing with Motion Tracking
University thesis
2013 · Author · shipped
My final-year thesis — spatial audio for VR, built before consumer VR existed.
Technologies: C#, C++, FMOD
Read the thesis (PDF, Spanish)
Related expertise: Audio & speech
Overview
This is a study on how our brain detects the direction of sounds in three-dimensional space.
The goal was to create a system that could convincingly simulate the spatial location of sounds in a virtual environment.
It has two main components: binaural processing to simulate how sounds reach the ears, and motion tracking to update the sound direction as the listener moves their head.
Binaural processing
Depending on the direction a sound comes from, it will reach each ear canal at slightly different times and with different intensities. The shape of the head and the ears themselves modify the sound, giving it a unique spatial signature that our brain is able to interpret for localization.
This spatial signature has been characterized as the Head Related Transfer Function (HRTF), which describes how an incoming sound is filtered by the listener's anatomy before reaching the eardrum.
The system uses these HRTFs to filter audio signals so they sound like they come from a particular direction.
I wrote that filter myself in C++ using FFTW. FMOD handled audio playback, and the rest of the system was written in C#.
Motion tracking
Once we have the filtering, we need to know the orientation of the listener's head in order to update the sound direction accordingly.
I chose to use a PlayStation Move controller for head tracking, as it provided a reliable and affordable solution at the time.
I took the controller's main board and mounted it on an audio headset. That allowed me to track the head's orientation and feed it into the audio filtering system.
Results
In the listening tests, a sound stayed fixed in place while the listener turned their head, and people could tell when it came from one side or from behind.
This was 2013, the year the first Oculus Rift development kit came out, and spatial audio for VR was still mostly a research topic. I came back to the same ideas years later, when I added positional audio to VR experiences. More in Audio & speech and VR, AR & spatial.
My involvement
- Designed and built the complete system as my final-year thesis.
- Wrote the real-time HRTF filter and integrated the head tracking.
- Ran listening tests to evaluate how well people could locate the sounds.