TL;DR
This paper introduces the ConferencingSpeech 2021 challenge, a new benchmark for far-field, multi-channel speech enhancement in video conferencing. It addresses the critical need for clear audio in remote communication by providing a comprehensive dataset and two distinct tasks: one focused on real-time enhancement with a single microphone array, and another exploring advanced techniques with multiple distributed arrays.
What Problem Does This Paper Address?
In today's increasingly remote world, video conferencing has become indispensable. However, the quality of audio in these calls is often hampered by a multitude of real-world acoustic challenges. These include background noise, reverberation within meeting rooms, the physical layout and acoustic properties of the room, and the placement and number of microphones. While significant progress has been made in speech enhancement, effectively tackling these issues in far-field, multi-channel scenarios, especially within the constraints of real-time video conferencing, remains a significant hurdle. This paper aims to stimulate research and development in this area by establishing a dedicated challenge and benchmark.
Methods
The ConferencingSpeech 2021 challenge is structured around two distinct tasks:
- Task 1: Real-time Single Microphone Array Enhancement: This task focuses on developing algorithms that can process speech from a single microphone array in real-time. Key constraints include a maximum frame length of 40ms and a real-time factor (RTF) of less than or equal to one on a standard processor. This emphasizes practical, low-latency applications.
- Task 2: Multi-channel Enhancement with Distributed Arrays: This task allows for more exploration, focusing on leveraging multiple, distributed microphone arrays to achieve superior speech quality. It is a non-real-time track, removing latency constraints and enabling participants to explore more complex algorithms.
The challenge also provides a baseline system based on a complex ratio mask (CRM) neural network. This system utilizes Short Time Fourier Transform (STFT) to extract magnitude and phase, then estimates inter-channel phase differences (IPD) to generate input features for a 3-layer LSTM. The output is a CRM that is applied to enhance the speech.
Datasets
The challenge introduces the ConferencingSpeech 2021 Challenge Database, recorded in real meeting rooms with real speakers to simulate realistic video conferencing scenarios. Key features of this database include:
- 12 rooms of varying sizes and decorative materials (categorized by the presence of glass walls).
- Recordings in both English and Chinese.
- Multiple microphone arrays (MAs) with three different geometric topologies:
- Linear MA with non-uniformly distributed 8 microphones.
- Circular MA with 16 microphones.
- Linear MA composed of two smaller linear MAs with uniformly distributed 8 microphones.
- Recordings are synchronized across all five MAs.
- The database includes semi-real recordings (simulated using playback and real speakers) and real recordings (from actual meetings).
The training set is designed under a "close training" condition, utilizing open-source datasets:
- Clean Speech: AISHELL-1, AISHELL-3, VCTK, and Librispeech (train-clean-360), totaling around 550 hours.
- Noise Set: MUSAN, Audioset, and real meeting room noises.
- Room Impulse Responses (RIRs): Simulated for various room sizes and microphone array placements.
Key Insights and Ideas
- The Importance of Realism: The challenge emphasizes the need for algorithms that perform well in realistic, far-field acoustic environments, moving beyond simulated or clean conditions.
- Dual Focus: Real-time vs. Quality: The two distinct tasks highlight the trade-off between real-time performance and achieving the absolute highest speech quality, catering to different application needs.
- Leveraging Multiple Arrays: Task 2 suggests that the spatial diversity offered by multiple distributed microphone arrays holds significant potential for improving speech enhancement performance.
- Subjective Evaluation is Key: The challenge relies on subjective evaluation (MOS, S-MOS, N-MOS) for final ranking, underscoring the perceptual aspect of speech quality.
Key Results
The paper presents results for the baseline system on the development simulation set. Across various microphone array configurations and tasks (single MA vs. multiple MA), the baseline system consistently improved objective metrics like PESQ, STOI, E-STOI, and Si-SNR compared to the noisy utterances. For instance, PESQ scores improved from around 1.5 to nearly 2.0, and Si-SNR increased from approximately 4.5 to over 9.0. The results suggest that the baseline system effectively enhances noisy speech, and the SNR selection strategy for multiple MAs showed a slight advantage over single MA models.
Limitations and Future Work
- Limited Training Data Sources: The "close training" condition restricts participants to the provided open-source datasets, which might not cover all possible acoustic scenarios.
- Subjective Evaluation Timeline: The final ranking and challenge results were scheduled for announcement in June 2021, indicating that the full impact of the challenge was yet to be determined at the time of publication.
- Future Work: The authors believe the challenge and its datasets will significantly promote research and development in far-field multi-channel speech enhancement. Future work will likely involve analyzing the submissions to identify successful strategies and further advance the state-of-the-art in this domain.
Summary updated