The question.
A voice assistant needs to handle turn-taking, accents, mixed-language speech, and real tasks without making callers wait through an entire pipeline.
The system design compares a modular speech-to-text → language model → speech pipeline with a realtime speech model. Calendar and knowledge tools operate behind explicit task boundaries.
The proposed approach.
- Stream partial speech recognition and measure end-of-turn detection
- Ground answers in a maintained knowledge source
- Confirm task details before committing a booking or record update
- Preserve a useful handoff summary when a caller requests a person
Target architecture.
Audio streamTurn detectionSpeech understandingAgent and toolsSpeech responseHuman handoff
The difficult parts.
- Nepali pronunciation and regional accents
- Barge-in without losing task context
- Network variation and provider failures
How to evaluate it.
- End-of-turn-to-first-audio latency distributions
- Task completion under interruptions and code-switching
- Handoff accuracy and transcript-supported answer quality
Planned deliverables.
- A bilingual conversation design kit
- A reference voice-session architecture
- A replayable audio evaluation set
Design principles.
Evaluate the conversation, not only its transcript.
A clean handoff is a successful product outcome.