Abstract
Deep learning-based models have greatly advanced the performance of speech enhancement (SE) systems. However, two problems remain unsolved, which are closely related to model generalizability to noisy conditions: (1) mismatched noisy condition during testing, i.e., the performance is generally sub-optimal when models are tested with unseen noise types that are not involved in the training data; (2) local focus on specific noisy conditions, i.e., models trained using multiple types of noises cannot optimally remove a specific noise type even though the noise type has been involved in the training data. In this paper, we propose a novel denoising autoencoder with a multi-branched encoder (termed DAEME) model to deal with these two problems. In the DAEME model, two stages are involved: training and testing. In the training stage, we build multiple component models to form a multi-branched encoder based on a decision tree (DSDT). The DSDT is built based on prior knowledge of speech and noisy conditions, where each component of the multi-branched encoder performs a particular mapping from noisy to clean speech along the branch in the DSDT. Finally, a decoder is trained on top of the multi-branched encoder. In the testing stage, noisy speech is first processed by each component model. The multiple outputs from these models are then integrated into the decoder to determine the final enhanced speech. Experimental results show that DAEME is superior to several baseline models in terms of objective evaluation metrics, automatic speech recognition results, and quality in subjective human listening tests.
Introduction
Speech enhancement (SE) is critical for improving the quality and intelligibility of voice signals in background noise, supporting applications from hearing aids to smart assistants. Despite the success of deep learning in SE, standard models often struggle with generalizability. Specifically, they face the "unseen noise" problem, where performance drops on novel noise types, and the "Jack of All Trades" trap, where a global model's performance on a specific noise is diluted by its broad training.
The DAEME (Denoising Autoencoder with Multi-branched Encoders) framework addresses these limitations by moving away from a "one-size-fits-all" approach. It partitions training data into specific "expert" branches based on acoustic attributes, allowing the system to capture both global patterns and local specialized knowledge.
Methodology
The DAEME architecture employs a "divide and conquer" strategy through a two-stage process: training and testing.
Dynamically-Sized Decision Tree (DSDT)
During training, data is hierarchically split using a DSDT. This tree incorporates:
- Utterance-level attributes (UAT): Initial splits are based on speaker gender (Male/Female) and Signal-to-Noise Ratio (High/Low SNR).
- Signal-level attributes (SAT): Leaf nodes are further refined by frequency components (e.g., low vs. high frequency) using spectral segmentation.
Multi-branched Encoder and Decoder
Each node in the DSDT corresponds to a component model (e.g., Bidirectional LSTMs) within a multi-branched encoder. Higher-level nodes capture general features, while lower-level nodes are trained to specialize in specific audio slices. In the testing stage, noisy audio passes through all component encoders. A final learned decoder then fuses these multiple "expert opinions" into a single, clean speech signal.
Experimental Setup
The model was evaluated on the Wall Street Journal (WSJ) Corpus and the TMHINT (Mandarin) dataset. To test true generalizability, the researchers used four noise types (Car, Pink, Street, Babble) that were entirely excluded from the training set.
Results and Discussion
DAEME consistently outperformed standard single-model BLSTMs across several metrics.
Key findings include:
- Objective Metrics: As shown in the PESQ results, DAEME achieved significantly higher quality scores, particularly in challenging "Babble" and "Street" environments. STOI metrics also indicated improved intelligibility in low-SNR conditions.
- ASR Performance: Enhanced audio led to a 28.07% relative reduction in Character Error Rate (CER) when processed by Google’s Speech-to-Text API.
- Knowledge Integration: Human-guided trees (using gender/SNR) outperformed unsupervised data-driven clustering, highlighting the value of prior acoustic knowledge.
Conclusion
The DAEME framework demonstrates that hierarchical specialization through multi-branched encoders significantly improves speech enhancement generalizability. While the approach introduces higher computational complexity, its superior performance in unseen noise conditions makes it a promising direction for robust SE systems. Future work will focus on automated tree design and optimizing for real-time latency in low-power devices.
Summary updated