Abstract
Automatic Speech Recognition (ASR) systems often suffer from performance degradation due to acoustic variability. While discriminative feature learning, such as Center Loss, can mitigate this by creating compact feature clusters, it typically requires frame-level labels. This paper introduces Temporal Multi-loss Fusion (TMF), a novel approach that integrates discriminative feature learning directly into sequence-based training (e.g., CTC) without requiring frame-level timing. By utilizing a Conditional Expected Center Loss (ECL) based on soft alignments from the forward-backward algorithm, the proposed method achieves significant Word Error Rate (WER) reductions, particularly in noisy environments.
Introduction
A major challenge in Automatic Speech Recognition (ASR) is maintaining robustness against acoustic variability, including background noise, reverberation, and speaker accents. Discriminative feature learning has proven effective in other domains, like computer vision, by ensuring that features of the same class are clustered closely in the embedding space while different classes are separated. However, applying these techniques to ASR is difficult because standard discriminative losses (e.g., Center Loss) require frame-level labels. Modern ASR systems favor sequence training objectives like Connectionist Temporal Classification (CTC), which operate on full transcripts without explicit timing information. This work bridges that gap by proposing a method to learn discriminative features within a sequence training framework.
Methodology
The researchers propose two levels of multi-loss fusion to enhance feature discriminability.
Framewise Multi-loss Fusion (FMF)
FMF combines standard Cross-Entropy (CE) loss with Center Loss. While effective at pulling features toward their respective class centers, it relies on "hard" frame-level labels, which are often unavailable or expensive to generate for large-scale ASR datasets.
Temporal Multi-loss Fusion (TMF)
TMF is the primary innovation, designed for sequence training where frame-level labels are absent. Instead of hard labels, TMF leverages the probabilistic "soft" alignments generated by the forward-backward algorithm in CTC training.
The core component is the Conditional Expected Center Loss (ECL). For each frame, the model calculates the probability of every possible label. The ECL then penalizes the feature representation based on its weighted distance to all potential class centers. This allows the network to learn compact, discriminative clusters while simultaneously optimizing the sequence recognition objective.
Experimental Setup
The models were trained on a large-scale, industrial dataset:
- Training Set: A noisy far-field corpus simulated from 2 million clean utterances (~1,840 hours), augmented with 15,000 room impulse responses and various background noises.
- Evaluation: Performance was measured on three test sets:
test_clean(real conversations),noise_seen(training-matched noise), andnoise_unseen(novel noise types).
Results and Discussion
The proposed TMF and FMF methods consistently outperformed baseline models across all test conditions.
Key findings include:
- Robustness to Unseen Noise: In the most challenging
noise_unseentest set, TMF achieved an 8.66% relative WER reduction over the standard CTC baseline. FMF achieved a 12.94% reduction over the CE baseline. - Generalization: By forcing the model to map acoustic signals into tight, distinct clusters, the resulting features are more resilient to noise-induced perturbations.
- Efficiency: The "centers" used for discriminative learning are only required during training. At inference time, the model operates without any additional computational overhead or parameters.
Conclusion
This work demonstrates that discriminative feature learning can be successfully integrated into sequence-based ASR training without the need for frame-level labels. The TMF approach, powered by the Conditional Expected Center Loss, significantly improves ASR robustness in noisy environments. Future research may explore the application of this method to modern Transformer-based architectures and its effectiveness in handling extreme speaker variability.
Summary updated