ICASSP 2022 Acoustic Echo Cancellation Challenge

Ross Cutler, Ando Saabas, Tanel Parnamaa, Marju Purin, Hannes Gamper, Sebastian Braun, Karsten Sørensen, Robert AichnerarXiv 2022 · arXiv

TL;DR

The ICASSP 2022 Acoustic Echo Cancellation (AEC) Challenge introduces massive new datasets—comprising over 10,000 real-world devices and environments—to push the boundaries of echo suppression in teleconferencing. The results highlight a significant performance gap in mobile scenarios and demonstrate that hybrid deep learning models are now the state-of-the-art, far surpassing traditional digital signal processing (DSP) methods.

What Problem Does This Paper Address?

Acoustic echo remains a primary culprit for poor call quality in platforms like Microsoft Teams or Zoom. While traditional DSP-based echo cancellers work well in controlled environments, they often fail in the "wild." Real-world challenges include non-linearities in the audio hardware, fast-changing acoustic conditions (like moving a laptop), and the difficult "double-talk" scenario where both parties speak simultaneously.

Furthermore, standard objective metrics like Echo Return Loss Enhancement (ERLE) and Perceptual Evaluation of Speech Quality (PESQ) correlate poorly with how humans actually perceive call quality in noisy, real-world settings. This paper details the third iteration of a challenge designed to provide the research community with the data and evaluation frameworks needed to solve these persistent issues, specifically focusing on the added complexity of mobile devices and high-fidelity 48 kHz audio.

Methods

The challenge organizers provided a baseline AEC method to serve as a starting point. This baseline adapts a noise suppression model into a recurrent neural network (RNN) with gated recurrent units (GRUs).

  • Input: Concatenated log power spectral features of the microphone and far-end signals.
  • Architecture: A stack of two GRU layers (322 nodes each) followed by a fully connected layer with a sigmoid activation.
  • Mechanism: The model predicts a spectral suppression mask that is point-wise multiplied with the microphone signal's magnitude spectrogram to suppress the echo.
  • Evaluation: The final challenge metric, $M$, is a weighted average of four subjective Mean Opinion Scores (MOS) and the Word Accuracy (WAcc) from a speech recognizer, ensuring that echo cancellation doesn't come at the cost of speech intelligibility.

Datasets

The authors released two substantial datasets to facilitate deep learning:

  1. Real Dataset: Captured via a massive crowdsourcing effort (Amazon Mechanical Turk), featuring over 50,000 recordings from 10,000+ unique environments and devices (Windows and Android). It includes diverse scenarios like echo path changes (moving the device) and sweep signals for estimating reverberation time (RT60).
  2. Synthetic Dataset: 10,000 scenarios generated by convolving clean speech with room impulse responses (RIRs) and adding background noise. RIRs were generated using Project Acoustics technology, covering RT60 ranges from 200ms to 1200ms.

![Fig.

  1. Distribution of reverberation time (RT60) across the datasets.](/figures/art_d3ae1d99ae8241658053ebfa5aeae9b0)

Key Insights and Ideas

  • The Metric Mismatch: The authors confirm that traditional metrics like ERLE (PCC 0.31) and PESQ (PCC 0.67) have low correlation with subjective human ratings. This justifies their push for AECMOS, a neural-network-based objective metric trained on human ground-truth ratings, which achieves much higher correlation (PCC > 0.8 for most scenarios).
  • Mobile is the New Frontier: Mobile devices are significantly more challenging than desktops. Because the loudspeaker is physically closer to the microphone, the signal-to-echo ratio is lower, leading to MOS scores that are roughly 0.75 lower in double-talk scenarios compared to desktops.
  • Hybrid vs. Pure Deep: The top-performing models are almost exclusively hybrid (combining linear AEC with DNNs) or pure deep learning models. Interestingly, the winning model used a linear AEC not to filter the audio directly, but as a feature to condition the DNN.

Key Results

The challenge saw 18 submissions, with the results summarized in the table below. The top models all utilized the Short-Time Fourier Transform (STFT) domain and performed simultaneous noise suppression and AEC.

Table 3. Comparison of the top 5 teams

A breakdown of the scores (Figure 3) shows that while many models achieved high scores in single-talk scenarios (MOS > 4.5), performance in double-talk (DT Echo DMOS) remains the primary differentiator between the top teams.

![Fig.

  1. Detailed AEC challenge results across different categories.](/figures/art_47e1b025f875422890bd787cefaead76)

Limitations and Future Work

The authors acknowledge that while AECMOS is a powerful tool, the version used for this challenge had not been trained on mobile or fullband (48 kHz) data, which is a priority for the next iteration. The significant performance drop on mobile devices indicates that current models still struggle with the high-intensity echoes and non-linearities inherent in smaller form factors. Future research will likely need to focus on more robust double-talk handling and model architectures specifically optimized for the constraints of mobile hardware.

Summary updated

Abstract

The ICASSP 2022 Acoustic Echo Cancellation Challenge is intended to stimulate research in acoustic echo cancellation (AEC), which is an important area of speech enhancement and still a top issue in audio communication. This is the third AEC challenge and it is enhanced by including mobile scenarios, adding speech recognition rate in the challenge goal metrics, and making the default sample rate 48 kHz. In this challenge, we open source two large datasets to train AEC models under both single talk and double talk scenarios. These datasets consist of recordings from more than 10,000 real audio devices and human speakers in real environments, as well as a synthetic dataset. We also open source an online subjective test framework and provide an online objective metric service for researchers to quickly test their results. The winners of this challenge are selected based on the average Mean Opinion Score achieved across all different single talk and double talk scenarios, and the speech recognition word acceptance rate.
BibTeX
@misc{cutler2022icassp,
  author = {Ross Cutler and Ando Saabas and Tanel Parnamaa and Marju Purin and Hannes Gamper and Sebastian Braun and Karsten Sørensen and Robert Aichner},
  title = {{ICASSP 2022 Acoustic Echo Cancellation Challenge}},
  year = {2022},
  eprint = {2202.13290},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.13290},
}
Figure