Introduction
In Automatic Speech Recognition (ASR), self-supervised learning (SSL) has emerged as a dominant paradigm for leveraging vast amounts of unlabelled audio data. The standard workflow involves pretraining a model on unlabelled data and subsequently adapting it to a specific task using a smaller amount of transcribed data. While most existing SSL research focuses on adaptation using frame-level objectives like Cross-Entropy (CE) or Connectionist Temporal Classification (CTC), this paper investigates the use of Lattice-Free Maximum Mutual Information (LFMMI)—a sequence-level discriminative criterion—for the adaptation phase.
The authors demonstrate that LFMMI adaptation significantly outperforms standard CE-based methods. Furthermore, they explore the robustness of this approach across domain shifts (conversational vs. read speech) and cross-lingual scenarios (English pretraining for low-resource languages like Swahili).
Methodology
The researchers employed a two-stage pipeline consisting of self-supervised pretraining followed by supervised adaptation.
Self-Supervised Pretraining
The pretraining phase utilized a Masked Acoustic Modeling (MAM) task, analogous to the BERT objective in natural language processing. A bidirectional Transformer was trained to reconstruct original 80-dimensional filterbank acoustic features from corrupted versions where segments were masked in time, frequency, or magnitude.
Supervised Adaptation with LFMMI
Instead of standard CE, the authors utilized flat-start LFMMI. This "end-to-end" version of the MMI criterion optimizes for the entire sequence and does not require pre-calculated alignments from a legacy system. Two primary adaptation strategies were tested:
- Feature Extraction: The pretrained Transformer weights are frozen, and its outputs serve as inputs for a separate Factorized Time-Delay Neural Network (TDNN-F).
- Fine-tuning (FT): The weights of both the pretrained Transformer and the TDNN-F are updated simultaneously.
Experimental Results
In-Domain Performance (Librispeech)
The models were pretrained on 960 hours of unlabelled Librispeech data and adapted using 100 hours of transcribed Librispeech data. As shown in the results below, the SSL-initialized models consistently outperformed supervised-only baselines.
A key insight from the Librispeech experiments is the significant boost in noise robustness. While relative Word Error Rate (WER) improvements on the "clean" test set were approximately 10%, the improvement on the "other" (noisy) test set reached a massive 35.3%. This suggests that the masking task during pretraining forces the model to learn robust, noise-invariant acoustic representations.
Domain Generalization (Switchboard)
To test domain shift, the authors evaluated the English-read-speech pretrained model on the Switchboard dataset (conversational English). The fine-tuned model achieved a WER of 9.4% on the SWBD portion of Hub5’00, representing a 10.8% relative improvement over the supervised baseline and outperforming standard TDNN-LSTM systems.
Cross-Lingual Generalization
The study also addressed whether pretraining on a high-resource language (English) benefits low-resource languages (Swahili and Tagalog).
In these cross-lingual scenarios, the choice of adaptation strategy proved critical. Using the pretrained model solely as a frozen feature extractor initially performed worse than a baseline trained from scratch. However, fine-tuning reversed this trend, allowing the model to recapture language-specific information while retaining the general acoustic primitives learned during English pretraining. This resulted in relative WER improvements of 4.3% for Swahili and 4.4% for Tagalog.
Conclusion
This research confirms that the choice of adaptation criterion is vital for SSL models. LFMMI consistently outperforms Cross-Entropy adaptation across various tasks. The combination of SSL pretraining and LFMMI adaptation provides substantial gains in noise robustness and generalizes effectively across different speaking styles and languages. Future work may involve combining this approach with multi-lingual pretraining and iterative semi-supervised decoding to further enhance performance in low-resource settings.
Summary updated