Lattice-Free Mmi Adaptation of Self-Supervised Pretrained Acoustic Models

Apoorv Vyas, Srikanth R. Madikeri, Hervé Bourlard

TL;DR

This paper demonstrates that adapting self-supervised pretrained acoustic models using Lattice-Free Maximum Mutual Information (LFMMI)—a sequence-level discriminative criterion—significantly outperforms standard cross-entropy-based adaptation. The authors show that this approach is highly robust, yielding substantial Word Error Rate (WER) reductions even when the target task involves a different domain (conversational vs. read speech) or a completely different language.

What Problem Does This Paper Address?

In Automatic Speech Recognition (ASR), self-supervised learning (SSL) has become a dominant paradigm for leveraging vast amounts of unlabelled audio data. Typically, a model is pretrained on unlabelled data and then "adapted" to a specific task using a smaller amount of transcribed data.

However, most existing SSL research focuses on adapting models using frame-level objectives like Cross-Entropy (CE) or Connectionist Temporal Classification (CTC). The authors argue that LFMMI, which optimizes for the entire sequence and is a staple of state-of-the-art HMM-DNN systems, is a better fit for this adaptation phase. They also set out to test if pretraining on one domain/language (e.g., English read speech) actually helps when the target task is vastly different (e.g., Swahili conversational speech).

Methods

The researchers employed a two-stage pipeline:

  1. Self-Supervised Pretraining: They used a Masked Acoustic Modeling (MAM) task, similar to BERT's objective in NLP. A bidirectional Transformer was trained to reconstruct original acoustic features (80-dimensional filterbanks) from corrupted versions where segments were masked in time, frequency, or magnitude.
  2. Supervised Adaptation with LFMMI: Instead of standard CE, they used flat-start LFMMI. This is an "end-to-end" version of the MMI criterion that doesn't require pre-calculated alignments from a legacy system.

They tested two adaptation strategies:

  • Feature Extraction: Freezing the pretrained Transformer and using its output as input for a separate TDNN-F (Factorized Time-Delay Neural Network).
  • Fine-tuning (FT): Updating the weights of the pretrained Transformer and the TDNN-F simultaneously.

Datasets

  • Pretraining: 960 hours of unlabelled Librispeech (read English speech).
  • Adaptation/Evaluation:
    • Librispeech (100h): In-domain adaptation.
  • Switchboard (300h): Domain shift (conversational English).
  • Babel (Swahili 38h, Tagalog 84h): Cross-lingual shift (low-resource languages).

Key Insights and Ideas

  • Sequence Objectives Matter: The "aha moment" is that the choice of adaptation criterion is crucial. For the same pretraining, LFMMI consistently beats Cross-Entropy (the "hybrid" approach).
  • Noise Robustness via Pretraining: Even when the amount of supervised data is identical, models initialized with SSL pretraining perform better on noisy test sets ("other" in Librispeech). This suggests the masking task forces the model to learn robust, noise-invariant representations.
  • The Fine-Tuning Necessity: In cross-lingual scenarios (English $\rightarrow$ Swahili), using the pretrained model as a frozen feature extractor actually performed worse than a baseline trained from scratch. However, fine-tuning reversed this, allowing the model to "recapture" language-specific information while benefiting from the general acoustic primitives learned during pretraining.
  • Domain Generalization: Pretraining on read speech (Librispeech) significantly boosts performance on conversational speech (Switchboard), proving that SSL captures fundamental acoustic properties that transcend speaking styles.

Key Results

The authors compared their LFMMI-adapted models against baselines trained only on supervised data:

  • Librispeech (100h): Relative WER improvements of 10% on the "clean" test set and a massive 35.3% on the "other" (noisy) test set.
  • Switchboard (300h): A relative WER improvement of 10.8% compared to the supervised baseline.
  • Babel (Cross-lingual): Relative WER improvements of 4.3% for Swahili and 4.4% for Tagalog.

Notably, their fine-tuned model on Switchboard achieved a WER of 9.4% (on the SWBD portion of Hub5'00), outperforming standard TDNN-LSTM baselines.

Limitations and Future Work

  • Comparison to wav2vec 2.0: While the paper references wav2vec 2.0, it doesn't provide a direct head-to-head comparison using the exact same LFMMI setup, which would be a valuable benchmark.
  • Architecture Specificity: The study focuses on Transformer and TDNN-F architectures; it's unclear if these gains translate identically to Conformer or other emerging architectures.
  • Future Directions: The authors suggest combining this SSL+LFMMI approach with multi-lingual pretraining and iterative decoding (semi-supervised learning) to further push the boundaries in low-resource settings.

Summary updated

BibTeX
@inproceedings{vyas2021lattice,
  author = {Apoorv Vyas and Srikanth R. Madikeri and Hervé Bourlard},
  title = {{Lattice-Free Mmi Adaptation of Self-Supervised Pretrained Acoustic Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=Rf-i6THlPN},
}
Figure