Abstract
Recent advancements in large language models (LLMs) have revolutionized various domains, bringing significant progress and new opportunities. Despite progress in speech-related tasks, LLMs have not been sufficiently explored in multi-talker scenarios. In this work, we present a pioneering effort to investigate the capability of LLMs in transcribing speech in multi-talker environments, following versatile instructions related to multi-talker automatic speech recognition (ASR), target talker ASR, and ASR based on specific talker attributes such as sex, occurrence order, language, and keyword spoken. Our approach utilizes WavLM and Whisper encoder to extract multi-faceted speech representations that are sensitive to speaker characteristics and semantic context. These representations are then fed into an LLM fine-tuned using LoRA, enabling the capabilities for speech comprehension and transcription. Comprehensive experiments reveal the promising performance of our proposed system, MT-LLM, in cocktail party scenarios, highlighting the potential of LLM to handle speech-related tasks based on user instructions in such complex settings.
Introduction
In real-world environments like crowded cafes or meetings, the "cocktail party problem"—where multiple people speak simultaneously—poses a significant challenge for Automatic Speech Recognition (ASR). Traditional ASR systems typically assume a single-speaker rule or require rigid, pre-recorded voiceprints to isolate a target. MT-LLM (Multi-Talker LLM) addresses this gap by leveraging the reasoning and instruction-following capabilities of Large Language Models. This framework allows users to interact with the system using natural language to specify who to transcribe based on attributes like sex, language, or specific keywords.
Methodology
MT-LLM bridges the gap between raw audio and text-based reasoning through a specialized architecture consisting of three main components:
- Dual Speech Encoders: The system employs two distinct processors. Whisper captures semantic context (the what), while WavLM extracts acoustic characteristics and speaker identity (the who).
- LLM Backbone: The model uses Llama-2-7b-chat as its central processing unit. To maintain efficiency, Low-Rank Adaptation (LoRA) is used for fine-tuning, updating only approximately 1% of the total parameters.
- Instruction-Based Tasks: The model is trained to follow six distinct instruction types, including multi-talker ASR, target-talker ASR via reference clips, keyword-tracing, and attribute-specific filtering (sex, order, or language).
Experimental Results
Evaluations were conducted using LibriSpeechMix and a custom En-De-Mixed dataset to test multilingual capabilities. MT-LLM demonstrated state-of-the-art performance across various multi-talker scenarios.
Key Insights
- Superior Accuracy: On the 2-speaker LibriSpeechMix benchmark, MT-LLM achieved a 5.2% Word Error Rate (WER), significantly outperforming the specialized D2V-Sidecar-DB model (7.5%).
- Instruction Reliability: In tasks such as Keyword-Tracing and Sex-Specific ASR, the model's performance was nearly identical to the theoretical "best-matching" upper bound, proving it rarely confuses speakers when given specific instructions.
- Multi-Task Synergy: Training on a variety of specific tasks (like keyword tracing) actually improved performance on basic multi-talker transcription. This suggests that learning to identify speaker attributes helps the model better parse the structure of overlapped speech.
- Multilingual Filtering: Despite limited German training data, the model successfully isolated German speakers from English ones with a WER of 21.8%.
Discussion and Future Work
While MT-LLM represents a major step forward, the authors identified areas for improvement. The model faced slight challenges with "Order-Specific" tasks (e.g., transcribing the second speaker), which might be mitigated by adding explicit positional embeddings for audio segments. Furthermore, while the current results rely on simulated mixtures, future work will focus on real-world audio containing background noise and reverb to enhance robustness.
Conclusion
MT-LLM demonstrates that LLMs can act as sophisticated filters in complex acoustic environments. By providing a text instruction, users can effectively steer the model's attention within high-dimensional audio embeddings, opening new avenues for interactive and context-aware speech recognition.
Summary updated