ICASSP 2023 Acoustic Echo Cancellation Challenge

Ross Cutler, Ando Saabas, Tanel Parnamaa, Marju Purin, Evgenii Indenbom, Nicolae-Catalin Ristea, Jegor Gužvin, Hannes Gamper, Sebastian Braun, Robert AichnerarXiv 2023 · arXiv

TL;DR

The ICASSP 2023 Acoustic Echo Cancellation (AEC) Challenge pushes the boundaries of real-time audio processing by introducing a personalized AEC track and slashing allowable latency to just 20ms. By providing massive open-source datasets and a robust subjective evaluation framework, the challenge reveals that while deep learning models are maturing, significant "headroom" remains for improving double-talk scenarios and near-end speech quality.

What Problem Does This Paper Address?

Acoustic echo—where a speaker's voice loops from a remote participant's loudspeaker back into their microphone—remains a primary source of poor call quality in teleconferencing. While traditional Digital Signal Processing (DSP) algorithms have handled this for decades, they often fail in the face of non-linearities (like cheap speaker distortion), fast-changing room acoustics, and "double-talk" (when both parties speak at once).

Furthermore, the industry lacks reliable ways to measure success. Standard objective metrics like ERLE (Echo Return Loss Enhancement) and PESQ (Perceptual Evaluation of Speech Quality) correlate poorly with how humans actually perceive quality in noisy, real-world environments. This paper details the fourth iteration of a global challenge designed to provide the data, metrics, and benchmarks needed to move AEC from synthetic lab conditions to messy, real-world applications.

Methods

The challenge was split into two tracks: Non-personalized AEC (general use) and Personalized AEC (which uses a 15-25 second "enrollment" clip of the near-end speaker to better distinguish their voice from the echo).

Key methodological shifts in 2023 included:

  • Stricter Latency: The total algorithmic and buffering latency was capped at 20ms, down from 40ms in previous years, to better reflect the requirements of real-time communication systems.
  • Subjective Evaluation: The authors utilized a "Third-party Listening Test" framework (based on ITU-T P.808/P.831). Unlike standard tests where you hear a single stream, this setup simulates a conversation where the listener hears the processed signal in one ear and a delayed version of the original "sidetone" in the other to detect annoying echoes.
  • Objective Proxy: They provided AECMOS, a neural network-based metric trained on human ratings, allowing researchers to iterate quickly without running expensive subjective tests.

The baseline model provided was a Recurrent Neural Network (RNN) using Gated Recurrent Units (GRUs) that predicts a spectral suppression mask based on the log power spectral features of the microphone and far-end signals.

Datasets

The authors released two substantial datasets to the community:

  1. Real Dataset: Over 50,000 recordings from 10,000+ unique audio devices and environments (captured via crowdsourcing). This includes various scenarios like echo path changes (moving the device during a call) and different room reverberation times (RT60).
  2. Synthetic Dataset: 10,000 scenarios generated by convolving clean speech with room impulse responses and adding non-linear loudspeaker distortions.
FIGURE 4. Distribution of reverberation time (RT60) for the synthetic dataset.

Key Insights and Ideas

  • The Efficiency Paradox: Interestingly, there was a negative correlation (PCC = -0.54) between model size and overall score. In this challenge, smaller, more efficient models tended to outperform massive ones, likely because they could better handle the strict 20ms latency requirement without sacrificing accuracy.
  • Metrics Matter: The paper confirms that ERLE and PESQ are essentially "blind" to subjective quality in these scenarios (Table 1). This highlights why the industry is shifting toward neural-network-based metrics like AECMOS.
  • Personalization Potential: While the first-place entry used personalized AEC, it only improved the final score by a tiny margin (0.002). This suggests that while the idea of using speaker identity to cancel echo is promising, we haven't yet fully cracked how to leverage that data effectively.
  • Real-World Complexity: Figure 3 illustrates the complex signal chain in modern devices, including built-in Audio DSP and Dynamic Range Compression (DRC), which AEC models must learn to "see through."
FIGURE 3. The custom recording application recorded the loopback and microphone signals.

Key Results

The challenge saw 20 entries, with top performers coming from ByteAudio, KuaiShou, and NWPU.

  • Latency vs. Quality: Microsoft’s internal model (MS-1) achieved a score of 0.856. Remarkably, this performance was statistically identical to the 2022 challenge winner, despite MS-1 operating at half the latency (20ms vs 40ms).
  • Leaderboard: The top-ranked models achieved Mean Opinion Scores (MOS) for near-end speech signal (ST NE SIG) around 4.2 to 4.3, indicating very high quality, though double-talk scenarios remained slightly lower.

Limitations and Future Work

The authors are transparent about the "headroom" remaining. While single-talk echo is largely solved, Double Talk Other Degradations (which includes audio cut-outs and distortions during simultaneous speech) still has a significant gap to perfection (0.64 headroom).

The ultimate goal for telecommunications is an end-to-end latency of less than 50ms to ensure natural conversation flow. To reach this, the authors suggest that future AEC components must aim for an algorithmic + buffering latency of less than 5ms, setting a high bar for the next generation of researchers.

Summary updated

Abstract

The ICASSP 2023 Acoustic Echo Cancellation Challenge is intended to stimulate research in acoustic echo cancellation (AEC), which is an important area of speech enhancement and is still a top issue in audio communication. This is the fourth AEC challenge and it is enhanced by adding a second track for personalized acoustic echo cancellation, reducing the algorithmic + buffering latency to 20ms, as well as including a full-band version of AECMOS. We open source two large datasets to train AEC models under both single talk and double talk scenarios. These datasets consist of recordings from more than 10,000 real audio devices and human speakers in real environments, as well as a synthetic dataset. We open source an online subjective test framework and provide an objective metric for researchers to quickly test their results. The winners of this challenge were selected based on the average mean opinion score (MOS) achieved across all scenarios and the word accuracy (WAcc) rate.
BibTeX
@misc{cutler2023icassp,
  author = {Ross Cutler and Ando Saabas and Tanel Parnamaa and Marju Purin and Evgenii Indenbom and Nicolae-Catalin Ristea and Jegor Gužvin and Hannes Gamper and Sebastian Braun and Robert Aichner},
  title = {{ICASSP 2023 Acoustic Echo Cancellation Challenge}},
  year = {2023},
  eprint = {2309.12553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.12553},
}
Figure