Real-time 3D on the web

Rendering, shaders and optimisation for 3D that has to run on whatever device the visitor brings.

Technologies: Three.js, WebGL, GLSL, React Three Fiber, drei

I've spent 12 years delivering 3D experiences in the browser across desktop, mobile and VR. Working under so many constraints teaches you a great deal about optimisation, content delivery and designing around what the hardware can actually do. The advantage is reach: a URL, nothing to install, and our experiences run on every device, OS and browser. (Even Internet Explorer!)

Hyperspace (2014)

The platform's whole proposition was that a 3D environment with avatars, interactive objects and conferencing should open from a link with no install. That put the rendering budget at the centre of the architecture rather than at the end of it.

Thousands of attendees at an event meant large environments with hundreds of objects. Dozens of booths with branding, hundreds of chairs, tables and plants. On top of the object loading and rendering pipeline — culling, LOD — I added automatic instancing with multi-texturing support. It meant one request to a chair's glb file instead of hundreds, and a much smaller texture memory footprint.

Texture memory also had a big impact on performance, especially on lower-end devices. Since we had no control over the assets users uploaded, I implemented a texture manager which downscaled textures on the server before they were ever delivered to the client. It worked automatically, picking the appropriate texture size for the hardware. Authors could also force specific textures to be resized at design time in our editor.

I created a performance monitor which governed what was rendered and how. It measured frame rate in real time and adapted dynamically, driving those budgets at runtime instead of fixing them up front. It regulated how far away objects would still be rendered or remain interactive — which saved us raycasts — along with shadows and physics.

Authors had access to a performance impact analysis tool they could run on demand, which measured drawcalls and texture memory, giving them a report on how the scene would perform before publishing.

On live events, each attendee is an avatar, and rendering them has taken up a large part of my work.

Expressive avatars are key to engaging in a virtual space. I have created several avatar animation systems where different animations play at once, built on the animation masking and layering I added to our custom fork of Three.js. Our avatars can walk or sit down while looking at another avatar, while talking or smiling, while holding an object in their hands. I created procedural animations (blinking, lipsync, head and shoulder orientation when looking at something), facial expressions and locomotion, drawing on more than 1,000 animations.

But there are limitations — regular hardware can't handle hundreds of fully textured animated avatars. It's something I fought with for years. Our rendering pipeline ended up implementing a wide variety of optimisation techniques, like animation FPS throttling, LODs or billboards. I created a custom GLSL shader which rendered low-poly geometry that resembled the original avatar very convincingly from a distance. It used no textures at all — six colours written straight into the shader, mapped to the regions people actually recognise: hair, skin, torso, legs. All these techniques allowed us to render more than 2,000 avatars in the same scene on regular consumer hardware.

Somebody AI (2025)

On this project, hyper-realistic avatars are built with Gaussian Splatting from a single photo. Our pipeline produces head-and-shoulders geometry made of 20k splats — enough for convincing visuals while still performing well in the browser.

Splats are bound to a FLAME face model whose expression parameters are mapped to Apple's ARKit blendshapes.

The avatar rendering engine is based on published research. My work was about making the avatar look more human and expressive. I added eye movement, facial expressions and lipsync.

For the facial expressions, I made a collection of ARKit poses which mapped to emotions. These emotions are triggered server-side, in real time, by NVIDIA's Audio2Emotion speech emotion recognition model. Lipsync runs the same way, through NVIDIA's Audio2Face.

Speak It Yourself VR (2019)

This VR project used 360 3D video mixed with UI. I modified our existing 360 panorama editor to support video and adapted our rendering pipeline to render it properly when the source was stereoscopic.

I also created a 3D UI system which mapped onto the platform's existing 2D interface. Users could interact with dialogue options by looking at them, pointing, or using speech recognition to activate them.

This work let the company land many more 360 video projects, and became a big differentiator for us as we were able to reach a wider range of clients.

Virtual Home Visit (2014)

Back in 2014, Three.js was at r69. This serious game was embedded as a SCORM module in an LMS, running on Internet Explorer.

One of the first challenges I faced was lighting: using more than one light source in the 3D scene was enough to drop the frame rate on the students' computers. Everything had to be baked in.

Students had to explore a two-storey house looking for fall risks and mark them. In the end we had to split the game into five levels so each scene could be rendered properly without issues.

Avatars were another interesting challenge. Internet Explorer (Safari mobile too, back then) had a limit on the number of bones that could influence a mesh. The trick was to split our skeleton into three parts (top, mid and bottom) and re-export every model. Then I would re-assemble the body parts at runtime. It was not ideal, but it let us ship thousands of avatar customisation items and animations. After some years I was able to transition from the old Three.js JSON format to glTF and render a single skeleton.

Where I've used it

View as Markdown