NeuralEcho: A Self-Attentive Recurrent Neural Network For Unified Acoustic Echo Suppression And Speech Enhancement

Meng Yu, Yong Xu, Chunlei Zhang, Shi-Xiong Zhang, Dong YuarXiv 2022 · DOI · arXiv

Introduction

In modern teleconferencing and hands-free communication, acoustic echo remains a significant challenge. While traditional solutions rely on linear filters, contemporary hardware often introduces non-linear distortions that degrade their performance. Researchers from Tencent AI Lab have developed NeuralEcho, a unified deep learning framework designed to jointly address acoustic echo cancellation (AEC), noise suppression (NS), and automatic gain control (AGC).

NeuralEcho is a compact, two-stage recurrent neural network (RNN) that leverages cross-channel second-order statistics and self-attention mechanisms. Despite having approximately 70% fewer parameters than state-of-the-art models like F-T-LSTM, it achieves superior performance in speech enhancement and recognition accuracy.

Problem Statement

Standard AEC systems face three primary hurdles in real-world environments:

  1. Non-linearity: Low-cost speakers and amplifiers warp signals, making the echo a non-linear transformation of the reference signal.
  2. Background Noise: Systems must distinguish between the target speaker's voice, background noise, and the echo simultaneously.
  3. Volume Consistency: Post-processing with AGC can inadvertently amplify residual noise if not properly integrated with the enhancement stage.

NeuralEcho addresses these by treating AEC, NS, and AGC as a unified task rather than a sequence of independent modules.

Proposed Method: The NeuralEcho Architecture

The core of NeuralEcho is a two-stage architecture that integrates traditional signal processing concepts into a neural framework.

Architecture of NeuralEcho showing the two-stage processing and speaker-aware FiLM layers.

Two-Stage Processing

  • Stage 1 (Initial Estimation): The model processes the microphone signal and far-end reference to calculate Channel, Temporal, and Frequency correlations. A Gated Recurrent Unit (GRU) encoder predicts complex ratio filters (cRF) to generate an initial echo estimate and a preliminary clean signal.
  • Stage 2 (Refinement): The model estimates covariance matrices for noise, echo, and target speech. These statistics feed into a Self-Attentive RNN. The Multi-Head Self-Attention (MHSA) mechanism allows the network to dynamically focus on relevant speech features across time frames.

Speaker Awareness and FiLM

To isolate the target user, NeuralEcho can incorporate a speaker embedding. Using Feature-wise Linear Modulation (FiLM), the model biases its filters toward the specific vocal characteristics of the enrolled speaker, enhancing the target voice while suppressing interferers.

Integrated Automatic Gain Control (AGC)

Unlike traditional systems that apply AGC as a post-processing step, NeuralEcho includes a dedicated branch for gain prediction. This branch is trained via multi-task learning alongside the main enhancement task.

Training framework for NeuralEcho, including the integrated Automatic Gain Control (AGC) branch.

By training the AGC jointly, the model learns to be more aggressive in noise suppression for segments that require significant volume boosting, preventing the amplification of artifacts.

Experimental Setup

The model was evaluated using extensive datasets, including AISHELL-2 for clean speech and the AEC-Challenge datasets for synthetic echo and non-linear distortions. Training involved 10,000 simulated Room Impulse Responses (RIRs) with reverberation times up to 0.6s. Testing conditions covered a wide range of Signal-to-Echo Ratios (SER) from -10dB to 10dB and Signal-to-Noise Ratios (SNR) from 0 to 40dB.

Key Results

NeuralEcho demonstrated significant improvements over the F-T-LSTM baseline across all metrics:

Metric F-T-LSTM (Baseline) NeuralEcho (Proposed) NeuralEcho + Speaker + AGC
SI-SDR (dB) 10.38 11.80 11.62
PESQ 2.88 2.98 3.01
WER (%) 15.86 14.42 13.20

The model improved the Signal-to-Distortion Ratio (SI-SDR) by 1.42 dB. Most notably, the Word Error Rate (WER) decreased from 15.86% to 13.20% when using the full configuration, representing a nearly 17% relative improvement.

Conclusion

NeuralEcho offers a highly efficient and effective solution for unified audio enhancement. By explicitly calculating second-order statistics and employing joint training for AGC, it outperforms larger models while remaining suitable for edge device deployment. Future work will explore extending this framework to multi-microphone beamforming arrays and further optimizing real-time latency for communication applications.

Summary updated

Abstract

Acoustic echo cancellation (AEC) plays an important role in the full-duplex speech communication as well as the front-end speech enhancement for recognition in the conditions when the loudspeaker plays back. In this paper, we present an all-deep-learning framework that implicitly estimates the second order statistics of echo/noise and target speech, and jointly solves echo and noise suppression through an attention based recurrent neural network. The proposed model outperforms the state-of-the-art joint echo cancellation and speech enhancement method F-T-LSTM in terms of objective speech quality metrics, speech recognition accuracy and model complexity. We show that this model can work with speaker embedding for better target speech enhancement and furthermore develop a branch for automatic gain control (AGC) task to form an all-in-one front-end speech enhancement system.
BibTeX
@misc{yu2022neuralecho,
  author = {Meng Yu and Yong Xu and Chunlei Zhang and Shi-Xiong Zhang and Dong Yu},
  title = {{NeuralEcho: A Self-Attentive Recurrent Neural Network For Unified Acoustic Echo Suppression And Speech Enhancement}},
  year = {2022},
  eprint = {2205.10401},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.10401},
}
Figure