You build the voice AI of Neybo, a children's device, and own it end to end: speech recognition, the model's answer, voice output, safety, latency, and the cost of a session.
The voice pipeline runs in the cloud as a streaming cascade: end-of-utterance detection -> speech-to-text -> LLM -> text-to-speech. The device handles wake word, voice activity detection, and basic functions. A child should see within 1-2 seconds of finishing a phrase that they were heard, so latency, interruptions, and long pauses are part of your daily design work.
You work with the Founding Engineer who builds the games, with the product team, and with the hardware product and legal colleagues who handle provider agreements. You own the pipeline, the safety cascade, the evaluation stand, and observability. In-game LLM behavior (character lines, game states, hints) belongs to the Founding Engineer, and you review it for safety and cost per line.
Voice pipeline
- Design and run the cloud cascade EOU -> STT -> LLM -> TTS with barge-in, token streaming into TTS, timeouts, and fallback between providers
- Hold end-to-end latency at 1-2 seconds from the end of the child's speech to sound from the speaker, measured at p50 and p90
- Tune end-of-utterance detection with an audio model plus a silence fallback of 1.5-3 seconds, since children pause for a long time
Quality and evaluation
- Build a repeatable evaluation stand on children's speech: golden dataset, held-out set, baselines, automatic runs on every model or prompt change
- Turn product requirements for understanding children's speech into metrics and thresholds on the stand
- Compare models on the chain "child's speech -> understanding -> answer" and prepare the recommendation on provider choice: safety, latency, cost, legal
Safety and content
- Build the safety cascade: refusals, redirection, risky topics, and rule updates without a firmware release
- Build answers from verified content (approved knowledge, constrained RAG, templates, tool calling), with dialogue states and transitions in code
- Review in-game LLM behavior for safety and cost per line
Observability and data
- Set up trace IDs, latency breakdown, session cost, and an error taxonomy together with the future CTO
- Implement data controls on the technical side: zero retention and training opt-out at providers, no storage of raw child audio
- 4+ years in ML/LLM engineering, including 1.5+ years with voice agents in production
- Built a streaming VAD/EOU -> STT -> LLM -> TTS cascade: barge-in, token streaming into TTS, timeouts, provider fallback. Know Pipecat, LiveKit Agents, or an equivalent, and can explain what you wrote yourself
- Optimized latency by percentiles: break end-to-end into EOU, STT, TTFT, TTS first byte, and network
- Built an eval harness: golden dataset, held-out set, baseline, automatic runs on model or prompt changes. Metrics such as WER, TTFT, EOU accuracy, tool-call success, LLM-as-judge calibrated on manual labels
- Built guardrails and safety classification for a sensitive audience: input and output filters, red-team sets, escalation
- Use structured output and tool calling as the basis of dialogue, with states and transitions in code. RAG over a curated knowledge base
- Python, async, backend with WebSocket/WebRTC, Docker, one of the major clouds. Observability: traces, cost per session
- Calculate unit economics: dollars per minute of STT/TTS, tokens per turn, caching
- Understand data retention and provider terms: zero retention, training opt-out, age restrictions in ToS
- English B2 or higher: vendors, certification bodies, documentation
- Children's products, edtech, or health: COPPA, PRIVO, kidSAFE
- Children's or accented speech in ASR: model selection, keyword boosting, normalization, fine-tuning small models (Whisper, Moonshine)
- Arabic in STT/TTS
- Work with devices: AEC, audio streaming from a Linux device, short-lived tokens
- Speech-to-speech models (OpenAI Realtime and similar) for comparison with the cascade
- Experience in a startup of up to 20 people
- A working evaluation stand on children's speech that the whole team can run and trust
- A safety cascade that handles critical cases by protocol and keeps false refusals on ordinary game lines low
- A cloud voice pipeline that the prototype's games run through
- End-to-end latency of 1-2 seconds from the end of the child's speech to sound from the speaker