TL;DR
This paper introduces LibriheavyMix, a massive 20,000-hour dataset designed to tackle the challenging problem of understanding speech in real-world scenarios with multiple overlapping speakers, reverberation, and far-field conditions. They also propose a baseline system and demonstrate the dataset's effectiveness for speech separation, recognition, and speaker diarization.
What Problem Does This Paper Address?
Modern speech processing systems excel at understanding clear, single-speaker audio, but struggle in complex, real-world environments. Think of a busy conference call, a noisy restaurant, or a lecture hall where multiple people might speak simultaneously, their voices echoing off walls, and the microphones are far from the speakers. Existing datasets often simplify these conditions by using only one speaker, near-field recordings, or lacking sufficient data for robust model training. This makes it difficult to build systems that can accurately transcribe "who said what and when" in these messy, everyday situations.
Methods
The authors tackle this by creating a new dataset and establishing baseline performance.
- Data Simulation: They leverage the large-scale Libriheavy ASR corpus, which includes rich transcriptions (punctuation, casing, context), and simulate multi-talker, reverberant speech. This involves:
- Overlapping Speech Simulation: Using statistical models derived from real meeting data to control the timing and overlap of different speakers' utterances.
- Reverberation Simulation: Employing a tool called FAST-RIR to realistically add room impulse responses to the dry speech signals, mimicking far-field acoustic environments.
- Baseline Systems: They implement and evaluate three core components of a speech understanding pipeline:
- Multi-Talker ASR: Using a Serialized Output Training (SOT) Conformer model that learns to predict a concatenated sequence of utterances, marked with a special "speaker change" token.
- Speech Separation: Employing the Conv-TasNet model, a well-established time-domain network for separating individual audio streams.
- Speaker Diarization: Utilizing the pre-trained pyannote.audio 3.1 system, a popular tool for identifying "who spoke when."
Datasets
- LibriheavyMix (New Dataset): This is the paper's primary contribution. It's a synthesized dataset comprising 20,000 hours of single-channel, reverberant, multi-talker speech. It's built upon the Libriheavy corpus and aims to provide a large-scale resource for speech separation, ASR, and speaker diarization. The dataset is available in various sizes (small, medium, large) and splits (train, dev, test).
- Libriheavy: Used as the source material for generating LibriheavyMix.
- AliMeeting: Statistics from this dataset were used to guide the simulation of realistic speaker overlap and pause durations in LibriheavyMix.
- WHAMR!: Used for evaluating the generalization capabilities of the proposed models and dataset.
- LibriSpeech: Used for pre-training the ASR models.
- wsj0-mix and WHAM!: Datasets that LibriheavyMix aims to improve upon in terms of scale and realism.
Key Insights and Ideas
- Scale Matters for Realism: The sheer volume of data (20,000 hours) in LibriheavyMix is crucial for training robust models that can handle the complexities of real-world speech, especially compared to smaller, existing datasets.
- Simulating Real-World Acoustics is Key: Incorporating both overlapping speech and reverberation is essential for bridging the gap between lab-controlled ASR and practical applications.
- A Unified Pipeline is Necessary: The paper demonstrates that these tasks (separation, ASR, diarization) are interconnected and can benefit from being addressed within a cohesive framework, especially when dealing with multi-talker scenarios.
- Generalization is a Challenge: Models trained on one dataset don't always perform well on others, highlighting the need for diverse and realistic training data like LibriheavyMix.
Key Results
- LibriheavyMix Performance: The paper shows that scaling up the training data for both ASR and speech separation models on LibriheavyMix leads to significant performance improvements, reducing word error rates and improving speaker counting accuracy.
- Speech Separation on LibriheavyMix: Conv-TasNet models trained on larger subsets of LibriheavyMix (e.g., "large2spk") consistently outperformed those trained on smaller subsets.
- ASR on LibriheavyMix: Similar trends were observed for the SOT ASR models, with larger training sets yielding better results, particularly for the challenging 4-speaker scenarios.
- Generalization to WHAMR!: Models trained on LibriheavyMix showed improved generalization to the WHAMR! dataset compared to models trained solely on WHAMR! or similar smaller datasets.
- Diarization Improvement: The speech separation module demonstrated a significant improvement in speaker diarization accuracy when used in conjunction with a diarization system, reducing the Diarization Error Rate (DER) by up to 11.51%.
Limitations and Future Work
- Synthesized Data Limitations: While LibriheavyMix is large and realistic, it is still synthesized. The authors acknowledge that real-world recordings might contain nuances not captured by simulation.
- Computational Cost: Training on such a massive dataset is computationally intensive, requiring significant resources.
- Future Work: The authors suggest further exploring the dataset for speaker-attributed ASR and more complex diarization tasks. They also imply that the dataset can be used to develop more advanced single-channel solutions that don't require multi-channel microphone arrays, making them more broadly applicable.
Summary updated