Learning discriminative features in sequence training without requiring framewise labelled data

Jun Wang, Dan Su, Jie Chen, Shulin Feng, Dongpeng Ma, Na Li, Dong Yu

Abstract

Automatic Speech Recognition (ASR) systems often suffer from performance degradation due to acoustic variability. While discriminative feature learning, such as Center Loss, can mitigate this by creating compact feature clusters, it typically requires frame-level labels. This paper introduces Temporal Multi-loss Fusion (TMF), a novel approach that integrates discriminative feature learning directly into sequence-based training (e.g., CTC) without requiring frame-level timing. By utilizing a Conditional Expected Center Loss (ECL) based on soft alignments from the forward-backward algorithm, the proposed method achieves significant Word Error Rate (WER) reductions, particularly in noisy environments.

Introduction

A major challenge in Automatic Speech Recognition (ASR) is maintaining robustness against acoustic variability, including background noise, reverberation, and speaker accents. Discriminative feature learning has proven effective in other domains, like computer vision, by ensuring that features of the same class are clustered closely in the embedding space while different classes are separated. However, applying these techniques to ASR is difficult because standard discriminative losses (e.g., Center Loss) require frame-level labels. Modern ASR systems favor sequence training objectives like Connectionist Temporal Classification (CTC), which operate on full transcripts without explicit timing information. This work bridges that gap by proposing a method to learn discriminative features within a sequence training framework.

Methodology

The researchers propose two levels of multi-loss fusion to enhance feature discriminability.

Framewise Multi-loss Fusion (FMF)

FMF combines standard Cross-Entropy (CE) loss with Center Loss. While effective at pulling features toward their respective class centers, it relies on "hard" frame-level labels, which are often unavailable or expensive to generate for large-scale ASR datasets.

Temporal Multi-loss Fusion (TMF)

TMF is the primary innovation, designed for sequence training where frame-level labels are absent. Instead of hard labels, TMF leverages the probabilistic "soft" alignments generated by the forward-backward algorithm in CTC training.

The core component is the Conditional Expected Center Loss (ECL). For each frame, the model calculates the probability of every possible label. The ECL then penalizes the feature representation based on its weighted distance to all potential class centers. This allows the network to learn compact, discriminative clusters while simultaneously optimizing the sequence recognition objective.

Experimental Setup

The models were trained on a large-scale, industrial dataset:

  • Training Set: A noisy far-field corpus simulated from 2 million clean utterances (~1,840 hours), augmented with 15,000 room impulse responses and various background noises.
  • Evaluation: Performance was measured on three test sets: test_clean (real conversations), noise_seen (training-matched noise), and noise_unseen (novel noise types).

Results and Discussion

The proposed TMF and FMF methods consistently outperformed baseline models across all test conditions.

Key findings include:

  • Robustness to Unseen Noise: In the most challenging noise_unseen test set, TMF achieved an 8.66% relative WER reduction over the standard CTC baseline. FMF achieved a 12.94% reduction over the CE baseline.
  • Generalization: By forcing the model to map acoustic signals into tight, distinct clusters, the resulting features are more resilient to noise-induced perturbations.
  • Efficiency: The "centers" used for discriminative learning are only required during training. At inference time, the model operates without any additional computational overhead or parameters.

Conclusion

This work demonstrates that discriminative feature learning can be successfully integrated into sequence-based ASR training without the need for frame-level labels. The TMF approach, powered by the Conditional Expected Center Loss, significantly improves ASR robustness in noisy environments. Future research may explore the application of this method to modern Transformer-based architectures and its effectiveness in handling extreme speaker variability.

Summary updated

Abstract

In this work, we try to answer two questions: Can deeply learned features with discriminative power benefit an ASR system's robustness to acoustic variability? And how to learn them without requiring framewise labelled sequence training data? As existing methods usually require knowing where the labels occur in the input sequence, they have so far been limited to many real-world sequence learning tasks. We propose a novel method which simultaneously models both the sequence discriminative training and the feature discriminative learning within a single network architecture, so that it can learn discriminative deep features in sequence training that obviates the need for presegmented training data. Our experiment in a realistic industrial ASR task shows that, without requiring any specific fine-tuning or additional complexity, our proposed models have consistently outperformed state-of-the-art models and significantly reduced Word Error Rate (WER) under all test conditions, and especially with highest improvements under unseen noise conditions, by relative 12.94%, 8.66% and 5.80%, showing our proposed models can generalize better to acoustic variability.
BibTeX
@inproceedings{wang2019learning,
  author = {Jun Wang and Dan Su and Jie Chen and Shulin Feng and Dongpeng Ma and Na Li and Dong Yu},
  title = {{Learning discriminative features in sequence training without requiring framewise labelled data}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8683088},
}
Figure