Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions

Lingwei Meng, Shujie Hu, Jiawen Kang, Zhaoqing Li, Yuejiao Wang, Wenxuan Wu, Xixin Wu, Xunying Liu, Helen Meng

TL;DR

MT-LLM is a new framework that empowers Large Language Models (LLMs) to navigate the "cocktail party problem"—transcribing overlapping speech from multiple people. By combining dual audio encoders with natural language instructions, the model can isolate specific speakers based on their sex, the language they speak, or even a specific keyword they mentioned, significantly outperforming previous specialized models in flexibility.

What Problem Does This Paper Address?

Most Automatic Speech Recognition (ASR) systems assume a "one person at a time" rule. In the real world—crowded cafes, meetings, or family dinners—people talk over each other. This is known as the Cocktail Party Problem.

While specialized models exist for multi-talker ASR, they are often rigid. They either transcribe everyone indiscriminately or require a pre-recorded "voiceprint" of a target speaker to work. There hasn't been a way to interact with an ASR system using natural language to specify who to listen to (e.g., "transcribe the person who just said 'strawberry'"). The authors address this gap by leveraging the reasoning and instruction-following capabilities of LLMs.

Methods

The researchers developed MT-LLM (Multi-Talker LLM), which bridges the gap between raw audio and text-based reasoning. The architecture consists of three main pillars:

  1. Dual Speech Encoders: Instead of relying on one audio processor, the authors use two:
  2. Whisper: Excellent at capturing semantic context (the what).
  3. WavLM: Excellent at capturing acoustic characteristics and speaker identity (the who).
  4. The Backbone: They use Llama-2-7b-chat as the central "brain." To keep the training efficient, they used LoRA (Low-Rank Adaptation), meaning they only updated about 1% of the model's parameters (76.6 million out of 7.55 billion).
  5. Instruction-Based Tasks: The model is trained to follow six distinct types of instructions:
  6. Multi-Talker ASR: Transcribe everyone in order.
  7. Target-Talker ASR: Transcribe a specific person given a 3-second reference clip.
  8. Keyword-Tracing: Transcribe the person who said a specific word.
  9. Sex-Specific: Transcribe only the male or female speakers.
  10. Order-Specific: Transcribe the first, second, or third person to speak.
  11. Target-Lingual: Transcribe only the person speaking a specific language (e.g., German).

Datasets

The authors used a mix of established and simulated datasets to train and test the model:

  • LibriSpeech (960h): Used as the primary source to simulate overlapping mixtures of 2 and 3 talkers.
  • CoVoST 2 (German subset): Integrated to support the "Target-Lingual" task, mixing English and German speakers.
  • LibriSpeechMix: The standard benchmark for evaluating multi-talker ASR.
  • En-De-Mixed: A "home-made" test set specifically designed to evaluate the model's ability to distinguish between English and German speakers in a mixture.

Key Insights and Ideas

  • Acoustic vs. Semantic Balance: The ablation study proved that Whisper alone isn't enough for multi-talker scenarios. Adding WavLM was crucial because its multi-layer features are more sensitive to the "texture" of different voices, which is necessary for separation.
  • Multi-Task Synergy: Interestingly, training the model on a variety of specific tasks (like keyword tracing or sex-specific ASR) actually improved its performance on the basic "transcribe everyone" task. This suggests that learning to identify speaker attributes helps the model better understand the structure of overlapped speech.
  • LLMs as Filters: The paper demonstrates that an LLM can act as a sophisticated "filter." By providing a text instruction, the user can steer the model's attention to a specific part of the high-dimensional audio embedding.

Key Results

MT-LLM showed impressive performance across the board, often beating specialized state-of-the-art models:

  • Multi-Talker ASR: On the 2-speaker LibriSpeechMix, MT-LLM achieved a 5.2% Word Error Rate (WER), outperforming the representative D2V-Sidecar-DB (7.5%).
  • Versatility: In tasks like Keyword-Tracing and Sex-Specific ASR, the model's performance was very close to the "best-matching" theoretical upper bound (e.g., 5.0% vs 4.8% for keyword tracing), proving it rarely gets confused about which speaker it was told to follow.
  • Multilingual Capability: Despite limited German training data, the model successfully isolated German speakers from English ones with a WER of 21.8%, showcasing its potential for complex, multi-lingual environments.

Limitations and Future Work

While a major step forward, the authors noted a few hurdles:

  • Order-Specific Challenges: The model struggled slightly more with "Order-Specific" tasks (transcribing the 1st vs 2nd speaker). The authors suggest that adding explicit positional embeddings for the audio segments could help the model keep track of time better.
  • Data Realism: The study relied on simulated mixtures. Real-world "cocktail party" audio includes background noise, reverb, and varying distances from the microphone, which will require more robust training data.
  • Scaling: The current model is a proof-of-concept for specific instructions. Future work aims to expand this into a more "universal" speech LLM that can handle an even broader range of tasks and languages.

Summary updated

Abstract

Recent advancements in large language models (LLMs) have revolutionized various domains, bringing significant progress and new opportunities. Despite progress in speech-related tasks, LLMs have not been sufficiently explored in multi-talker scenarios. In this work, we present a pioneering effort to investigate the capability of LLMs in transcribing speech in multi-talker environments, following versatile instructions related to multi-talker automatic speech recognition (ASR), target talker ASR, and ASR based on specific talker attributes such as sex, occurrence order, language, and keyword spoken. Our approach utilizes WavLM and Whisper encoder to extract multi-faceted speech representations that are sensitive to speaker characteristics and semantic context. These representations are then fed into an LLM fine-tuned using LoRA, enabling the capabilities for speech comprehension and transcription. Comprehensive experiments reveal the promising performance of our proposed system, MT-LLM, in cocktail party scenarios, highlighting the potential of LLM to handle speech-related tasks based on user instructions in such complex settings. The code, model, and samples are available at https://github.com/cuhealthybrains/MT-LLM.
BibTeX
@inproceedings{meng2025large,
  author = {Lingwei Meng and Shujie Hu and Jiawen Kang and Zhaoqing Li and Yuejiao Wang and Wenxuan Wu and Xixin Wu and Xunying Liu and Helen Meng},
  title = {{Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10887642},
}

Loading related content...

Figure