ICASSP 2021 Acoustic Echo Cancellation Challenge: Datasets, Testing Framework, and Results

Kusha Sridhar, Ross Cutler, Ando Saabas, Tanel Parnamaa, Markus Loide, Hannes Gamper, Sebastian Braun, Robert Aichner, Sriram SrinivasanarXiv 2020 · arXiv

TL;DR

The ICASSP 2021 Acoustic Echo Cancellation (AEC) Challenge addresses the performance gap between synthetic lab models and real-world teleconferencing. By open-sourcing a massive dataset of 2,500+ real environments and a robust subjective testing framework, the authors demonstrate that deep learning models significantly outperform traditional digital signal processing (DSP) methods in complex, noisy scenarios.

What Problem Does This Paper Address?

Acoustic echo—that distracting loopback of your own voice during a video call—remains a primary source of poor audio quality. While researchers have developed many AEC algorithms, most are trained and tested on synthetic datasets where the training and testing data come from the same distribution. When these models meet the "wild"—diverse hardware, background noise, and varying room acoustics—their performance often plummets.

Furthermore, the industry has long relied on objective metrics like Echo Return Loss Enhancement (ERLE) and Perceptual Evaluation of Speech Quality (PESQ). However, these metrics correlate poorly with actual human perception in realistic, noisy environments. This paper introduces a challenge designed to move the field toward more representative datasets and reliable subjective evaluation.

Methods

The challenge organizers provided a baseline AEC method to set the bar. They adapted a noise suppression model into a recurrent neural network (RNN) using Gated Recurrent Units (GRUs). This model takes log power spectral features from both the microphone and the far-end signal to predict a spectral suppression mask.

The evaluation process was particularly rigorous. Instead of relying on automated scores, the winners were determined by Mean Opinion Scores (MOS) collected through a crowdsourced subjective testing framework based on the ITU-T P.808 standard. The audio pipeline used for testing (shown below) included stages for Automatic Gain Control (AGC) and noise suppression to ensure a fair comparison of the AEC components themselves.

![Fig.

  1. The custom recording application recorded the loopback and microphone signals.](/figures/art_33589a6cf6ff41baa21f100e382d4779)

Datasets

The authors introduced two significant new datasets:

  1. Real Dataset: A massive crowdsourced effort via Amazon Mechanical Turk, capturing recordings from over 2,500 unique audio devices and environments. It covers scenarios like single-talk, double-talk (both parties speaking), and echo path changes (moving the device).
  2. Synthetic Dataset: 10,000 scenarios (100 hours of audio) using LibriVox speech and noise from Audioset and Freesound. It includes simulated room impulse responses and nonlinear distortions to mimic cheap loudspeakers.

The diversity of the real-world environments is captured in the distribution of reverberation times (RT60), which ranged from 200ms to 1200ms.

![Fig.

  1. Distribution of reverberation time (RT60).](/figures/art_93b1e3663d924c8891b19545f558e64b)

Key Insights and Ideas

  • The Metric Gap: The authors confirmed that ERLE and PESQ have low correlation with subjective human tests in real-world conditions. ERLE, for instance, is only truly appropriate for quiet rooms and single-talk scenarios.
  • Scenario Trade-offs: A model that excels at suppressing echo during single-talk might struggle during "double-talk," where it might accidentally suppress the local speaker's voice (distortion). The challenge results highlighted that top-performing models must balance these conflicting requirements.
  • Deep Learning Dominance: The results proved that deep learning and hybrid models (combining DSP and AI) can significantly outperform traditional DSP models, even under the strict latency requirements (≤40ms) necessary for real-time communication.

Key Results

Seventeen teams submitted models. The final standings were based on an average of MOS across four categories: Near-end single talk, Far-end single talk echo, Double-talk echo, and Double-talk other degradations.

  • Winner: Team 21 emerged as the clear winner with an overall MOS of 4.11.
  • Statistical Significance: While the first-place team was a clear outlier, the differences between the 2nd through 5th place teams were not statistically significant (as shown in the ANOVA test in Figure 6), leading the organizers to use computational complexity as a tie-breaker for those spots.

![Fig.

  1. Final results of the challenge.](/figures/art_f4e7c29d42f5443899664eeba5cb3160)

![Fig.

  1. P-values of ANOVA test of the top 5 teams.](/figures/art_6c373cd947284c4fa55dfcf9b48b114d)

Limitations and Future Work

The authors acknowledge that their current "Overall MOS" is a simple unweighted mean of different scenarios. Future research is needed to determine a more sophisticated weighting system that better reflects how users prioritize different types of audio degradations. Additionally, while deep learning models won, there is still significant room for improvement in handling the most difficult double-talk scenarios without introducing audible distortion.

Summary updated

Abstract

The ICASSP 2021 Acoustic Echo Cancellation Challenge is intended to stimulate research in the area of acoustic echo cancellation (AEC), which is an important part of speech enhancement and still a top issue in audio communication and conferencing systems. Many recent AEC studies report good performance on synthetic datasets where the train and test samples come from the same underlying distribution. However, the AEC performance often degrades significantly on real recordings. Also, most of the conventional objective metrics such as echo return loss enhancement (ERLE) and perceptual evaluation of speech quality (PESQ) do not correlate well with subjective speech quality tests in the presence of background noise and reverberation found in realistic environments. In this challenge, we open source two large datasets to train AEC models under both single talk and double talk scenarios. These datasets consist of recordings from more than 2,500 real audio devices and human speakers in real environments, as well as a synthetic dataset. We open source two large test sets, and we open source an online subjective test framework for researchers to quickly test their results. The winners of this challenge will be selected based on the average Mean Opinion Score (MOS) achieved across all different single talk and double talk scenarios.
BibTeX
@misc{sridhar2020icassp,
  author = {Kusha Sridhar and Ross Cutler and Ando Saabas and Tanel Parnamaa and Markus Loide and Hannes Gamper and Sebastian Braun and Robert Aichner and Sriram Srinivasan},
  title = {{ICASSP 2021 Acoustic Echo Cancellation Challenge: Datasets, Testing Framework, and Results}},
  year = {2020},
  eprint = {2009.04972},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.04972},
}
Figure