Abstract
Moshi is a first-of-its-kind, full-duplex speech-text foundation model designed for natural, real-time conversation. By transitioning from traditional cascaded pipelines (ASR-LLM-TTS) to a unified multi-stream architecture, Moshi achieves a practical latency of 200ms—faster than the average human response time. This approach allows the model to naturally handle interruptions, backchanneling, and non-linguistic nuances like emotion and tone.
Introduction
Traditional voice assistants often suffer from an "uncanny valley" effect due to their cascaded architecture. These systems typically rely on a sequence of Voice Activity Detection (VAD), Automatic Speech Recognition (ASR), a Large Language Model (LLM), and Text-to-Speech (TTS). This pipeline introduces significant latency, loses emotional information during the text bottleneck, and enforces a rigid turn-based interaction. Moshi addresses these challenges by treating spoken dialogue as a single, continuous modeling task.
Architecture and Methodology
The core of Moshi is a unified architecture that models parallel streams: the user’s audio and the system’s audio. This allows the model to "listen" and "talk" simultaneously, effectively removing the concept of a "speaker turn."
The Mimi Codec
Moshi utilizes Mimi, a neural audio codec that converts raw 24kHz audio into discrete tokens via Residual Vector Quantization (RVQ). Mimi is fully causal and distills semantic information from self-supervised models (WavLM) directly into its tokens. This enables the model to process linguistic content directly from audio without a separate transcription step.
Inner Monologue
To maintain linguistic coherence, Moshi employs an "Inner Monologue" mechanism. Before generating audio tokens, the model predicts time-aligned text tokens as a prefix. This allows the system to leverage the reasoning capabilities of its 7B-parameter text backbone, Helium, while outputting raw audio in real-time.
Training and Data
Moshi's development involved several stages of pre-training and fine-tuning across text and audio modalities. The Helium backbone was pretrained on 2.1 trillion tokens of public English data. Audio pretraining utilized 7 million hours of raw audio, followed by fine-tuning on the Fisher dataset (2,000 hours) for natural turn-taking and 20,000 hours of synthetic dialogue scripts.
As shown in the hyper-parameter table, the model utilizes a Temporal Transformer with a model dimension of 4096 and a Depth Transformer for acoustic generation. The training process incorporates specific acoustic and text delays to stabilize generation and improve performance.
Key Results
- Latency: Moshi achieves a theoretical latency of 160ms and a practical latency of 200ms, surpassing the average human response time of approximately 230ms.
- Performance: The model outperforms existing speech-text models like SpeechGPT on spoken question-answering benchmarks.
- Full-Duplex Capability: Qualitative evaluations show Moshi can handle complex conversational dynamics, including graceful handling of interruptions and backchanneling.
Discussion and Future Work
While Moshi represents a breakthrough in real-time dialogue, several challenges remain. Training on audio data led to a slight degradation in the model's raw factual knowledge compared to text-only versions. Additionally, ensuring safety through robust generative watermarking is a priority for future iterations. Moshi moves AI interaction closer to the fluid, emotional reality of human conversation, providing a powerful open-source tool for the research community.
Summary updated