TriNet: stabilizing self-supervised learning from complete or slow collapse

Lixin Cao, Jun Wang, Ben Yang, Dan Su, Dong Yu

Abstract

Self-supervised learning (SSL) for speech recognition often suffers from informational collapse, where models learn degenerate representations. TriNet introduces a novel triple-branch architecture that incorporates a frozen teacher model as a high-level anchor. This stabilization mechanism prevents both complete and slow collapse, leading to 33% faster convergence and a 6% relative reduction in Word Error Rate (WER) compared to the state-of-the-art Data2vec framework.

Introduction

Self-supervised learning (SSL) has revolutionized speech processing by enabling models to learn from vast amounts of unlabeled data. However, a persistent challenge in SSL is "informational collapse"—a state where the model outputs constant or low-variance vectors regardless of the input. While existing methods like Data2vec use student-teacher bootstrapping to mitigate this, they remain susceptible to "slow collapse," particularly in speech where temporal correlations are high. TriNet addresses these instabilities by introducing a third "frozen" branch that acts as a stable anchor for the learning process.

The Challenge of Model Collapse

In speech SSL, models typically predict masked segments of audio. Because adjacent audio frames are highly similar, models can easily find "shortcuts" that minimize loss without learning meaningful features. This often manifests as a gradual loss of representation variance over time. Current solutions, such as contrastive learning or complex architectural constraints, can be computationally expensive or difficult to tune and stabilize.

Methodology

TriNet, named after the stable three-legged "trivet," utilizes a triple-branch architecture to process audio data:

  1. Student Branch: A standard Conformer encoder that processes perturbed (masked) audio input.
  2. EMA Teacher Branch: An Exponential Moving Average (EMA) of the student’s weights, providing mid-level latent targets for the student to regress toward.
  3. Frozen Teacher Branch: A pre-trained, fixed ASR model that generates high-level "pseudo-target" vectors from the original, unmasked input.
The TriNet architecture featuring three branches: Student, EMA Teacher, and Frozen Teacher.

The core innovation is the "Anchor" mechanism. While the Student and EMA Teacher may drift together toward a collapsed state, the Frozen Teacher provides a fixed reference point in a high-level pseudo-class space. This forces the student to maintain distinct representations for different audio classes, effectively preventing embedding degeneracy. The model optimizes a dual loss: a structural loss ($L_{struc}$) for latent embedding regression and a regularization loss ($L_{reg}$) for predicting high-level pseudo-labels.

Experimental Results

TriNet was evaluated on standard benchmarks including Librispeech (LS-960h) and the massive Libri-light (LL-21k) dataset. The results demonstrate that TriNet consistently outperforms existing state-of-the-art (SOTA) models.

Performance comparison of TriNet against state-of-the-art models on Librispeech (LS-960h) and Libri-light (LL-21k).

Key findings include:

  • Word Error Rate (WER): TriNet achieved a 6.06% relative reduction in WER on Librispeech and a 5.89% reduction on the larger Libri-light dataset compared to Data2vec.
  • Efficiency: The training process is significantly more stable, allowing TriNet to reach optimal performance in approximately 66% of the time required by Data2vec.
  • Representation Quality: t-SNE visualizations confirm that TriNet produces more distinct clusters for audio classes than its predecessors, avoiding the "smearing" of representations seen in collapsed models.

Conclusion

TriNet provides a robust solution to the problem of informational collapse in speech SSL. By integrating a frozen teacher as a stable anchor, the framework achieves superior performance, faster convergence, and greater scalability. While it currently depends on a pre-trained teacher, the student model eventually surpasses the teacher's performance, proving it learns superior representations rather than merely copying. Future work will explore heterogeneous teacher architectures and offline pseudo-label generation to further optimize the training pipeline.

Summary updated

BibTeX
@inproceedings{cao2023trinet,
  author = {Lixin Cao and Jun Wang and Ben Yang and Dan Su and Dong Yu},
  title = {{TriNet: stabilizing self-supervised learning from complete or slow collapse}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10094725},
}
Figure