Audio & speech

Sound and voice — spatial audio, adaptive scoring, text-to-speech at scale, and lipsync generated in real time.

Technologies: Wwise, Flash, FMOD

I've been passionate about audio since I was a teenager, and specialised in video and audio engineering in my university degree. I've designed sound effects, delivered speech for millions of users and built real-time speech-to-speech conversational avatars.

Spirit Fencer (2012)

I created a library of 50+ effects for spells and attacks, and built an audio manager in ActionScript to play them back within a tight memory budget. Only the effects belonging to the characters in a given combat were pre-loaded, and a fixed buffer of tracks capped how many could sound at once.

Given the game was Flash (SWF) and its size had to be below 10MB, I applied many optimisation techniques (mono vs stereo, sampling rate and perceived quality, looping and encoding issues...).

Binaural Processing with Motion Tracking (2013)

As my final-year thesis, I wanted to bring together audio processing, motion tracking and VR.

I used FMOD as the main engine for audio. I created an audio filter in C++ which would apply HRTF responses in real time based on the physical geometry of the head and its orientation. The results were overwhelmingly positive: I was able to lock a sound in place in virtual space while the listener moved their head. Listeners felt the sound was coming from one side, or from behind.

I shared my work with some research groups and years later I experimented with positional audio in VR experiences using the same principles.

Tu Casa Es La Mía (2016)

I wanted to experiment with procedural audio on this small project.

For sound effects, I created an audio processor linked to Wwise with sample collections (several variations for the same effect) so, for example, a ship shooting repeatedly wouldn't feel like the same sample playing multiple times. The engine would vary the sample to play, its pitch and panning randomly so the sound environment would feel more realistic.

I created a short song for the game. The music was layered, game state was monitored and the music would adapt in real time. If the game was quiet (no one was attacking), only the main melody would be heard. When ships started attacking, drums and bass would come in. When the action increased, the synths would come in to give the user a feeling of thrill.

Hyperspace (2014)

Audio, and speech in particular, was a very important part of this authoring platform. Almost all the experiences our clients built had conversational avatars that used text-to-speech to speak their lines. I had to maximise device support and provide a robust experience for users, as the conversations were crucial for their engagement.

I used ResponsiveVoice for text-to-speech and created a tool to estimate the duration of every dialogue line, so even if the speech fails or isn't available, captions appear on screen for as long as the line would have taken, maintaining a natural flow in the conversation.

I added lipsync to our avatars, so that when users spoke in a VR meeting, their characters' lips followed.

Another challenge was handling microphone and speaker permissions in the browser. Over the years browsers and devices have become increasingly restrictive on how to play sound. I had a permission handler that would detect the type of device and display different popups and volume previews only when they were really needed.

I built speech recognition into the platform. Users could interact with the experience by using their voice, and actually communicate with AI-driven avatars in real time.

I experimented with positional audio in live events, for both sound effects and speech.

ResponsiveVoice (2014)

This text-to-speech library was designed to work across any device and OS. It used the browser's or device's native TTS engine where available, and fell back to our server-side voice service, which returned audio files.

The biggest challenge was chunking: speech generation had a character limit, so I had to create an algorithm that splits big paragraphs into smaller sections. Each section would be requested sequentially and played in order, so timing had to be exact to maintain a natural cadence.

Somebody AI (2025)

This project uses speech-to-speech models for real-time conversations with AI-driven avatars.

We do a lot of work under the hood. Speech is not transmitted continuously: voice activity is detected and only real speech is sent through to our servers. In our backend voice service, the response from the speech-to-speech model is processed in real time — the audio stream is split into chunks — to generate lipsync and facial expressions through NVIDIA's Audio2Emotion and Audio2Face models.

Speech and animation metadata are transmitted back to our client to play in sync, bringing the avatar to life.

Tools

Where I've used it

View as Markdown