TL;DR
The ICASSP 2021 Acoustic Echo Cancellation (AEC) Challenge addresses the performance gap between synthetic lab models and real-world teleconferencing. By open-sourcing a massive dataset of 2,500+ real environments and a robust subjective testing framework, the authors demonstrate that deep learning models significantly outperform traditional digital signal processing (DSP) methods in complex, noisy scenarios.
What Problem Does This Paper Address?
Acoustic echo—that distracting loopback of your own voice during a video call—remains a primary source of poor audio quality. While researchers have developed many AEC algorithms, most are trained and tested on synthetic datasets where the training and testing data come from the same distribution. When these models meet the "wild"—diverse hardware, background noise, and varying room acoustics—their performance often plummets.
Furthermore, the industry has long relied on objective metrics like Echo Return Loss Enhancement (ERLE) and Perceptual Evaluation of Speech Quality (PESQ). However, these metrics correlate poorly with actual human perception in realistic, noisy environments. This paper introduces a challenge designed to move the field toward more representative datasets and reliable subjective evaluation.
Methods
The challenge organizers provided a baseline AEC method to set the bar. They adapted a noise suppression model into a recurrent neural network (RNN) using Gated Recurrent Units (GRUs). This model takes log power spectral features from both the microphone and the far-end signal to predict a spectral suppression mask.
The evaluation process was particularly rigorous. Instead of relying on automated scores, the winners were determined by Mean Opinion Scores (MOS) collected through a crowdsourced subjective testing framework based on the ITU-T P.808 standard. The audio pipeline used for testing (shown below) included stages for Automatic Gain Control (AGC) and noise suppression to ensure a fair comparison of the AEC components themselves.

Datasets
The authors introduced two significant new datasets:
- Real Dataset: A massive crowdsourced effort via Amazon Mechanical Turk, capturing recordings from over 2,500 unique audio devices and environments. It covers scenarios like single-talk, double-talk (both parties speaking), and echo path changes (moving the device).
- Synthetic Dataset: 10,000 scenarios (100 hours of audio) using LibriVox speech and noise from Audioset and Freesound. It includes simulated room impulse responses and nonlinear distortions to mimic cheap loudspeakers.
The diversity of the real-world environments is captured in the distribution of reverberation times (RT60), which ranged from 200ms to 1200ms.

Key Insights and Ideas
- The Metric Gap: The authors confirmed that ERLE and PESQ have low correlation with subjective human tests in real-world conditions. ERLE, for instance, is only truly appropriate for quiet rooms and single-talk scenarios.
- Scenario Trade-offs: A model that excels at suppressing echo during single-talk might struggle during "double-talk," where it might accidentally suppress the local speaker's voice (distortion). The challenge results highlighted that top-performing models must balance these conflicting requirements.
- Deep Learning Dominance: The results proved that deep learning and hybrid models (combining DSP and AI) can significantly outperform traditional DSP models, even under the strict latency requirements (≤40ms) necessary for real-time communication.
Key Results
Seventeen teams submitted models. The final standings were based on an average of MOS across four categories: Near-end single talk, Far-end single talk echo, Double-talk echo, and Double-talk other degradations.
- Winner: Team 21 emerged as the clear winner with an overall MOS of 4.11.
- Statistical Significance: While the first-place team was a clear outlier, the differences between the 2nd through 5th place teams were not statistically significant (as shown in the ANOVA test in Figure 6), leading the organizers to use computational complexity as a tie-breaker for those spots.


Limitations and Future Work
The authors acknowledge that their current "Overall MOS" is a simple unweighted mean of different scenarios. Future research is needed to determine a more sophisticated weighting system that better reflects how users prioritize different types of audio degradations. Additionally, while deep learning models won, there is still significant room for improvement in handling the most difficult double-talk scenarios without introducing audible distortion.
Summary updated