AI and LLM

Company-wide agentic workflows, AI-driven avatars and the pipelines behind them — from branching dialogue trees to real-time voice models.

Technologies: OpenAI, Pipecat, Ollama, PyTorch, Python

I use agents in my workflow and provide AI-driven experiences to thousands of users.

Embedding AI-driven agents into immersive experiences has proven to be challenging, and much more interesting than expected.

On the coding side, I've been designing specs for coding agents since 2024. I've created agents for running CI, creating Jira tickets and GitHub issues. I stay up to date and experiment weekly.

Somebody AI (2025)

Somebody AI delivers ultra-realistic conversational avatars with speech-to-speech capability.

I created a platform that allows clients to design and embed their own agents, by customising every aspect of them: appearance, voice, goals, behaviour and limitations. The platform features a prompt assistant that helps customers define how the agent behaves. Depending on the type of agent, it helps them shape the prompt, drawing on our years of experience defining directives, rubrics and guardrails.

On the technical side, the frontend captures audio and sends it to a server, which acts as a gateway to the LLM provider. The audio stream coming back is used to generate lipsync and expression, which are sent to the client. It supports tool calling both on the server and in the host page. I designed the integration with Pipecat, which allows the product to use almost any model on the market.

On the graphics side, the avatar generation pipeline is built on PyTorch and generates Gaussian splats from a single photo. I deployed the pipeline and generated all the templates for the product.

ChatMapper (2015)

This non-linear dialogue authoring tool allowed me to create engaging experiences where users would want to spend hours. Branching conversations feel very comfortable, because the user has the illusion of free choice: they do pick the answer (which was designed for them to pick in the first place) and have the feeling of accomplishment, without having to come up with the response on their own.

With AI agents we can let the user talk freely to a virtual character without scripting anything, which seems like a big leap, but carries many problems. Making sure the agent stays on track and is accurate about the content you provide is very challenging. Giving it its own particular voice or personality is just not achievable today. There's no guarantee the learner hits the teaching point. And it has cost and latency attached.

After a few years creating prompts for AI-driven agents, I still plan and design character dialogues with ChatMapper in mind, and still think branching conversations are superior for a scripted, enjoyable experience.

Hyperspace (2014)

We've had conversational bots since our beginning in 2014. We used ChatMapper to deliver branching conversations.

Before ChatGPT existed, we were already using AI for our bots. Our platform integrated Google's DialogFlow to drive conversations. So, when LLMs were released, we just plugged them in.

I integrated OpenAI for AI bots in 2023 through their completions API. We've had thousands of participants interacting with AI bots in our platform since then, and we learned a lot about prompts over the years.

I've experimented with different architectures to build augmented pipelines, like LangChain (with Ollama), n8n or Pipecat, using a variety of models and tools. We've had our avatars powered by n8n workflows with RAG for corporate clients as it allows for quick experimentation.

I even started building an assistant that would allow authors to decorate their 3D scenes using their voice.

I've also done speech-to-speech, image and video generation. We integrated with services like Tavus to embed live-generated avatars in our scenes.

On the coding side, I've designed workflows for agents on CI (an agent can submit PRs, tag for deployment, monitor CI and submit issues on its own), Jira ticket management, and documentation. I've also defined workflows where agents can communicate with each other while still being supervised by a human.

I've fine-tuned our coding agents' directives on each product, so we control how far an agent can go on its own while implementing features. They include specific guardrails: changes to core parts of the codebase require a report and a project lead review to authorise.

Where I've used it

View as Markdown