Speak It Yourself VR
SIYVR
2019 · CTO & Lead Developer · shipped
VR training for spoken English, built on 360° video. Learners stand inside a real conversation, answer out loud, and speech recognition checks what they say.
Technologies: JavaScript, Three.js, HTML5, WebXR
Related expertise: Real-time 3D on the web, VR, AR & spatial
Overview
Speak It Yourself is a tool for practising spoken English, made for the language training company SIYVR. Instead of reading a dialogue from a textbook, the learner watches a 360° video of people talking around them. When it's their turn, they choose what to say and say it out loud, and speech recognition checks the answer and gives feedback straight away.
The videos can be stereoscopic, so in a VR headset the learner feels like they are standing in the scene. Alongside the conversations, there were immersive grammar courses for use in class.
How it was built
It ran on Hyperspace. The platform already had a 360° panorama editor, which I extended to play video and to render stereoscopic sources correctly.
The learner picks their answer in a HUD that follows their gaze. It works with hand controllers, or with gaze alone when there are none, and it moves with the head without causing motion sickness.
This work let the company take on many more 360° video projects afterwards.
My involvement
- Built the 360° video playback, for monoscopic and stereoscopic sources.
- Built the in-VR interface for reading and choosing dialogue options.
- Integrated speech-to-text to assess the learner's spoken answers.
More about this project, by area: