TL;DR
The ICASSP 2023 Acoustic Echo Cancellation (AEC) Challenge pushes the boundaries of real-time audio processing by introducing a personalized AEC track and slashing allowable latency to just 20ms. By providing massive open-source datasets and a robust subjective evaluation framework, the challenge reveals that while deep learning models are maturing, significant "headroom" remains for improving double-talk scenarios and near-end speech quality.
What Problem Does This Paper Address?
Acoustic echo—where a speaker's voice loops from a remote participant's loudspeaker back into their microphone—remains a primary source of poor call quality in teleconferencing. While traditional Digital Signal Processing (DSP) algorithms have handled this for decades, they often fail in the face of non-linearities (like cheap speaker distortion), fast-changing room acoustics, and "double-talk" (when both parties speak at once).
Furthermore, the industry lacks reliable ways to measure success. Standard objective metrics like ERLE (Echo Return Loss Enhancement) and PESQ (Perceptual Evaluation of Speech Quality) correlate poorly with how humans actually perceive quality in noisy, real-world environments. This paper details the fourth iteration of a global challenge designed to provide the data, metrics, and benchmarks needed to move AEC from synthetic lab conditions to messy, real-world applications.
Methods
The challenge was split into two tracks: Non-personalized AEC (general use) and Personalized AEC (which uses a 15-25 second "enrollment" clip of the near-end speaker to better distinguish their voice from the echo).
Key methodological shifts in 2023 included:
- Stricter Latency: The total algorithmic and buffering latency was capped at 20ms, down from 40ms in previous years, to better reflect the requirements of real-time communication systems.
- Subjective Evaluation: The authors utilized a "Third-party Listening Test" framework (based on ITU-T P.808/P.831). Unlike standard tests where you hear a single stream, this setup simulates a conversation where the listener hears the processed signal in one ear and a delayed version of the original "sidetone" in the other to detect annoying echoes.
- Objective Proxy: They provided AECMOS, a neural network-based metric trained on human ratings, allowing researchers to iterate quickly without running expensive subjective tests.
The baseline model provided was a Recurrent Neural Network (RNN) using Gated Recurrent Units (GRUs) that predicts a spectral suppression mask based on the log power spectral features of the microphone and far-end signals.
Datasets
The authors released two substantial datasets to the community:
- Real Dataset: Over 50,000 recordings from 10,000+ unique audio devices and environments (captured via crowdsourcing). This includes various scenarios like echo path changes (moving the device during a call) and different room reverberation times (RT60).
- Synthetic Dataset: 10,000 scenarios generated by convolving clean speech with room impulse responses and adding non-linear loudspeaker distortions.
Key Insights and Ideas
- The Efficiency Paradox: Interestingly, there was a negative correlation (PCC = -0.54) between model size and overall score. In this challenge, smaller, more efficient models tended to outperform massive ones, likely because they could better handle the strict 20ms latency requirement without sacrificing accuracy.
- Metrics Matter: The paper confirms that ERLE and PESQ are essentially "blind" to subjective quality in these scenarios (Table 1). This highlights why the industry is shifting toward neural-network-based metrics like AECMOS.
- Personalization Potential: While the first-place entry used personalized AEC, it only improved the final score by a tiny margin (0.002). This suggests that while the idea of using speaker identity to cancel echo is promising, we haven't yet fully cracked how to leverage that data effectively.
- Real-World Complexity: Figure 3 illustrates the complex signal chain in modern devices, including built-in Audio DSP and Dynamic Range Compression (DRC), which AEC models must learn to "see through."
Key Results
The challenge saw 20 entries, with top performers coming from ByteAudio, KuaiShou, and NWPU.
- Latency vs. Quality: Microsoft’s internal model (MS-1) achieved a score of 0.856. Remarkably, this performance was statistically identical to the 2022 challenge winner, despite MS-1 operating at half the latency (20ms vs 40ms).
- Leaderboard: The top-ranked models achieved Mean Opinion Scores (MOS) for near-end speech signal (ST NE SIG) around 4.2 to 4.3, indicating very high quality, though double-talk scenarios remained slightly lower.
Limitations and Future Work
The authors are transparent about the "headroom" remaining. While single-talk echo is largely solved, Double Talk Other Degradations (which includes audio cut-outs and distortions during simultaneous speech) still has a significant gap to perfection (0.64 headroom).
The ultimate goal for telecommunications is an end-to-end latency of less than 50ms to ensure natural conversation flow. To reach this, the authors suggest that future AEC components must aim for an algorithmic + buffering latency of less than 5ms, setting a high bar for the next generation of researchers.
Summary updated