Moshi: a speech-text foundation model for real-time dialogue

Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, Neil ZeghidourarXiv 2024 · DOI · arXiv

Abstract

Moshi is a first-of-its-kind, full-duplex speech-text foundation model designed for natural, real-time conversation. By transitioning from traditional cascaded pipelines (ASR-LLM-TTS) to a unified multi-stream architecture, Moshi achieves a practical latency of 200ms—faster than the average human response time. This approach allows the model to naturally handle interruptions, backchanneling, and non-linguistic nuances like emotion and tone.

Introduction

Traditional voice assistants often suffer from an "uncanny valley" effect due to their cascaded architecture. These systems typically rely on a sequence of Voice Activity Detection (VAD), Automatic Speech Recognition (ASR), a Large Language Model (LLM), and Text-to-Speech (TTS). This pipeline introduces significant latency, loses emotional information during the text bottleneck, and enforces a rigid turn-based interaction. Moshi addresses these challenges by treating spoken dialogue as a single, continuous modeling task.

Architecture and Methodology

The core of Moshi is a unified architecture that models parallel streams: the user’s audio and the system’s audio. This allows the model to "listen" and "talk" simultaneously, effectively removing the concept of a "speaker turn."

Figure 1: The Moshi architecture featuring the Mimi codec, Helium Temporal Transformer, and the multi-stream modeling approach.

The Mimi Codec

Moshi utilizes Mimi, a neural audio codec that converts raw 24kHz audio into discrete tokens via Residual Vector Quantization (RVQ). Mimi is fully causal and distills semantic information from self-supervised models (WavLM) directly into its tokens. This enables the model to process linguistic content directly from audio without a separate transcription step.

Inner Monologue

To maintain linguistic coherence, Moshi employs an "Inner Monologue" mechanism. Before generating audio tokens, the model predicts time-aligned text tokens as a prefix. This allows the system to leverage the reasoning capabilities of its 7B-parameter text backbone, Helium, while outputting raw audio in real-time.

Training and Data

Moshi's development involved several stages of pre-training and fine-tuning across text and audio modalities. The Helium backbone was pretrained on 2.1 trillion tokens of public English data. Audio pretraining utilized 7 million hours of raw audio, followed by fine-tuning on the Fisher dataset (2,000 hours) for natural turn-taking and 20,000 hours of synthetic dialogue scripts.

Figure 4: Hyper-parameters for the Temporal and Depth Transformers across various training stages.

As shown in the hyper-parameter table, the model utilizes a Temporal Transformer with a model dimension of 4096 and a Depth Transformer for acoustic generation. The training process incorporates specific acoustic and text delays to stabilize generation and improve performance.

Key Results

  • Latency: Moshi achieves a theoretical latency of 160ms and a practical latency of 200ms, surpassing the average human response time of approximately 230ms.
  • Performance: The model outperforms existing speech-text models like SpeechGPT on spoken question-answering benchmarks.
  • Full-Duplex Capability: Qualitative evaluations show Moshi can handle complex conversational dynamics, including graceful handling of interruptions and backchanneling.

Discussion and Future Work

While Moshi represents a breakthrough in real-time dialogue, several challenges remain. Training on audio data led to a slight degradation in the model's raw factual knowledge compared to text-only versions. Additionally, ensuring safety through robust generative watermarking is a priority for future iterations. Moshi moves AI interaction closer to the fluid, emotional reality of human conversation, providing a powerful open-source tool for the research community.

Summary updated

Abstract

We introduce Moshi, a speech-text foundation model and full-duplex spoken dialogue framework. Current systems for spoken dialogue rely on pipelines of independent components, namely voice activity detection, speech recognition, textual dialogue and text-to-speech. Such frameworks cannot emulate the experience of real conversations. First, their complexity induces a latency of several seconds between interactions. Second, text being the intermediate modality for dialogue, non-linguistic information that modifies meaning -- such as emotion or non-speech sounds -- is lost in the interaction. Finally, they rely on a segmentation into speaker turns, which does not take into account overlapping speech, interruptions and interjections. Moshi solves these independent issues altogether by casting spoken dialogue as speech-to-speech generation. Starting from a text language model backbone, Moshi generates speech as tokens from the residual quantizer of a neural audio codec, while modeling separately its own speech and that of the user into parallel streams. This allows for the removal of explicit speaker turns, and the modeling of arbitrary conversational dynamics. We moreover extend the hierarchical semantic-to-acoustic token generation of previous work to first predict time-aligned text tokens as a prefix to audio tokens. Not only this "Inner Monologue" method significantly improves the linguistic quality of generated speech, but we also illustrate how it can provide streaming speech recognition and text-to-speech. Our resulting model is the first real-time full-duplex spoken large language model, with a theoretical latency of 160ms, 200ms in practice, and is available at https://github.com/kyutai-labs/moshi.
BibTeX
@misc{defossez2024moshi,
  author = {Alexandre Défossez and Laurent Mazaré and Manu Orsini and Amélie Royer and Patrick Pérez and Hervé Jégou and Edouard Grave and Neil Zeghidour},
  title = {{Moshi: a speech-text foundation model for real-time dialogue}},
  year = {2024},
  eprint = {2410.00037},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.00037},
}
Figure