Abstract
Sequence-to-sequence (S2S) modeling is becoming a popular paradigm for automatic speech recognition (ASR) because of its ability to jointly optimize all the conventional ASR components in an end-to-end (E2E) fashion. This report investigates the ability of E2E ASR from standard close-talk to far-field applications by encompassing entire multichannel speech enhancement and ASR components within the S2S model. There have been previous studies on jointly optimizing neural beamforming alongside E2E ASR for denoising. It is clear from both recent challenge outcomes and successful products that far-field systems would be incomplete without solving both denoising and dereverberation simultaneously. This report uses a recently developed architecture for far-field ASR by composing neural extensions of dereverberation and beamforming modules with the S2S ASR module as a single differentiable neural network and also clearly defining the role of each subnetwork. The original implementation of this architecture was successfully applied to the noisy speech recognition task (CHiME-4), while we applied this implementation to noisy reverberant tasks (DIRHA and REVERB). Our investigation shows that the method achieves better performance than conventional pipeline methods on the DIRHA English dataset and comparable performance on the REVERB dataset. It also has additional advantages of being neither iterative nor requiring parallel noisy and clean speech data.
Introduction
When interacting with far-field speech recognition systems, such as smart speakers, audio quality is significantly degraded by two primary factors: background noise and reverberation (sound reflections). Traditional Automatic Speech Recognition (ASR) systems typically address these issues using a modular "pipeline" approach. In such systems, a dereverberation module first removes echoes, followed by a beamformer that utilizes multiple microphones to spatially filter noise, and finally, the processed audio is passed to the ASR engine.
However, this decoupled approach has significant drawbacks. Each component is often optimized independently, meaning the enhancement modules do not necessarily optimize for the specific features required by the ASR engine. Furthermore, many state-of-the-art enhancement methods are computationally expensive (iterative) or require parallel data—recordings of the same speech in both clean and noisy environments—which is difficult to obtain for real-world scenarios.
Architecture and Methods
The proposed system integrates dereverberation, beamforming, and ASR into a single, unified differentiable neural network. This allows the entire pipeline to be jointly optimized using backpropagation based solely on the final speech recognition objective.
The architecture consists of three primary stages:
- Neural Dereverberation (WPE Subnetwork): A neural version of Weighted Prediction Error (WPE) is employed. Instead of traditional iterative mathematical optimization, a subnetwork predicts a power spectrum mask to estimate and subtract late reflections.
- Neural Beamforming (MVDR Subnetwork): The dereverberated signal enters a Minimum Variance Distortionless Response (MVDR) subnetwork. Masking networks for speech and noise calculate the statistics required for spatial filtering. An attention-based mechanism is used to automatically select the best reference microphone signal.
- E2E ASR Module: The enhanced signal is converted into Log Mel Filterbank features and processed by a hybrid CTC-attention encoder-decoder model to predict text.
By treating the entire system as a single network, the enhancement modules learn to "clean" audio specifically to minimize the Word Error Rate (WER), without requiring explicit clean audio targets during training.
Experimental Setup
The model was trained using 2-channel simulated data from the REVERB challenge and clean data from the Wall Street Journal (WSJ) corpus. Evaluation was conducted on two distinct datasets:
- REVERB: 8-channel real and simulated sets, serving as a standard benchmark.
- DIRHA-WSJ: 6-channel real recordings from a living room. This represents a "mismatch" condition, as the acoustic environment differs significantly from the training data.
Results and Discussion
The experimental results demonstrate that joint optimization is highly effective, particularly in mismatched conditions. The E2E model generalized better to real-world DIRHA data than traditional pipeline methods, despite being trained only on REVERB simulations.
Key performance highlights include:
- DIRHA Dataset: The proposed E2E model achieved a 29.1% WER, outperforming the best pipeline method (30.7% WER), a 5.2% relative improvement.
- REVERB Real Data: The system achieved 8.7% WER (Near) and 12.4% WER (Far), a more than 50% relative improvement over the E2E baseline without enhancement.
- Interpretability: Despite the end-to-end nature, the modular design allows for intermediate audio inspection. Spectrogram analysis confirms that the WPE subnetwork reduces temporal smearing while the beamformer effectively suppresses background noise.
Conclusion
This investigation confirms that integrating multichannel enhancement directly into an E2E ASR framework provides superior robustness in reverberant and mismatched environments. The approach eliminates the need for parallel training data and iterative processing, making it a promising candidate for real-time far-field applications. Future work will focus on bridging the remaining performance gap in extreme far-field conditions and exploring more diverse microphone geometries during training.
Summary updated