Once-for-All Sequence Compression for Self-Supervised Speech Models

Hsuan-Jui Chen, Yen Meng, Hung-yi Lee

TL;DR

Researchers have developed a "Once-for-All" (OFA) framework for self-supervised speech models that allows a single model to support a continuous range of sequence compression rates. By adjusting a single parameter at inference time, users can trade off computational efficiency for performance across diverse tasks—from high-resolution speech recognition to ultra-compressed speaker identification—without the need to retrain the model for each specific use case.

What Problem Does This Paper Address?

Self-supervised learning (SSL) models like HuBERT and wav2vec 2.0 have revolutionized speech processing, but they come with a heavy "computational tax." Because speech is sampled at high frequencies, the resulting sequences are incredibly long, leading to high memory usage and slow inference.

While researchers have used "subsampling" (reducing the number of frames) to speed things up, most existing methods use a fixed compression rate. This is problematic because different speech tasks have different needs:

  • Automatic Speech Recognition (ASR) requires fine-grained temporal detail to distinguish between similar-sounding words.
  • Speaker Identification (SID) or Intent Classification (IC) can often work with much coarser, "utterance-level" information.

Currently, if you want an optimized model for both ASR and SID, you might have to train two separate models with different compression settings. This paper proposes a way to train one model that fits all these needs.

Methods

The authors build their framework on top of DistilHuBERT, a smaller "student" model trained to mimic a larger "teacher" model. To achieve flexible compression, they integrate a Continuous Integrate-and-Fire (CIF) module.

The key innovation is the $\alpha$ modification strategy:

  1. The $\alpha$ Module: This part of the network predicts "weights" ($\alpha$) for each input frame. When the accumulated weights reach a certain threshold, the model "fires," producing one output representation for that segment.
  2. The Once-for-All Parameter ($\lambda$): They introduce a scalar $\lambda \in [0, 2)$ that modifies these weights during training and inference.
  3. $\lambda < 1$ (Up-scaling): Increases the number of output frames (lower compression) for tasks requiring high resolution.
  4. $\lambda > 1$ (Down-scaling): Decreases the number of output frames (higher compression) for tasks where efficiency is priority.
  5. Adaptive Learning: Beyond manual selection, they show that $\lambda$ can be treated as a learnable parameter. During downstream task tuning, the model can automatically "learn" the optimal compression rate for that specific task.

Datasets

  • Pre-training: The model was pre-trained on the standard LibriSpeech 960-hour dataset.
  • Guidance: To help the CIF module learn where to "fire," they used unsupervised phonetic segments generated from a model trained on LibriSpeech 100-hour data.
  • Evaluation: The framework was tested on the SUPERB benchmark, covering nine diverse tasks:
  • Content: Phoneme Recognition (PR), ASR, Query by Example (QbE), Speech Translation (ST).
  • Speaker: Speaker Identification (SID).
  • Semantics/Intent: Keyword Spotting (KS), Intent Classification (IC), Slot Filling (SF).
  • Emotion: Emotion Recognition (ER).

Key Insights and Ideas

  • The "Sweet Spot" is Task-Dependent: The paper confirms that while ASR performance degrades quickly as you compress the sequence, tasks like Keyword Spotting (KS) are incredibly resilient. You can compress the sequence until each frame represents nearly one second of audio and still maintain ~92% accuracy.
  • Continuous vs. Discrete: Unlike previous "on-demand" models that only offered a few fixed choices (e.g., 1x or 2x compression), this OFA approach allows for a "smooth sweep" across the entire spectrum of frame periods (from 20ms to 960ms).
  • Zero-Cost Trade-offs: The OFA model performs just as well as "specialized" models trained at a single fixed compression rate. You get the flexibility of multiple models with the training cost of one.

Key Results

  • Massive Computation Reduction: For utterance-level tasks, the model achieved a 90.6% reduction in Multiply-Accumulate operations (MACs) in the Transformer layers by using a 960ms frame period.
  • Competitive Performance: On the SUPERB benchmark, the OFA model matched or exceeded the performance of previous state-of-the-art compressed models (like the variable-length subsampling version of HuBERT) across almost all tasks.
  • Successful Adaptive Learning: The adaptive $\lambda$ mechanism successfully converged to optimal compression rates for PR, SF, and IC without requiring a manual grid search, proving that the model can "self-optimize" for new tasks.

Limitations and Future Work

  • Dependency on Phonetic Guidance: The current CIF implementation relies on pre-extracted phonetic segments for guidance during training. While these can be obtained via unsupervised methods, it adds a layer of preprocessing complexity.
  • Performance Floor: At extreme compression rates (near 1 second per frame), performance on complex tasks like Speech Translation and Slot Filling drops significantly. Finding ways to retain semantic meaning at ultra-high compression remains a challenge.
  • Future Directions: The authors suggest that this "once-for-all" philosophy could be extended beyond sequence length to other dimensions of the model, such as the number of layers or hidden dimensions.

Summary updated

Abstract

The sequence length along the time axis is often the dominant factor of the computation in speech processing. Works have been proposed to reduce the sequence length for lowering the computational cost in self-supervised speech models. However, different downstream tasks have different tolerance of sequence compressing, so a model that produces a fixed compressing rate may not fit all tasks. In this work, we introduce a once-for-all (OFA) sequence compression framework for self-supervised speech models that supports a continuous range of operating compressing rates. The framework is evaluated on various tasks, showing marginal degradation compared to the fixed compressing rate variants with a smooth performance-efficiency trade-off. We further explore adaptive compressing rate learning, demonstrating the ability to select task-specific preferred frame periods without needing a grid search.
BibTeX
@inproceedings{chen2023once,
  author = {Hsuan-Jui Chen and Yen Meng and Hung-yi Lee},
  title = {{Once-for-All Sequence Compression for Self-Supervised Speech Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095025},
}

Loading related content...

Figure