Abstract
The sequence length along the time axis is often the dominant factor of the computation in speech processing. Works have been proposed to reduce the sequence length for lowering the computational cost in self-supervised speech models. However, different downstream tasks have different tolerance of sequence compressing, so a model that produces a fixed compressing rate may not fit all tasks. In this work, we introduce a once-for-all (OFA) sequence compression framework for self-supervised speech models that supports a continuous range of operating compressing rates. The framework is evaluated on various tasks, showing marginal degradation compared to the fixed compressing rate variants with a smooth performance-efficiency trade-off. We further explore adaptive compressing rate learning, demonstrating the ability to select task-specific preferred frame periods without needing a grid search.
Introduction
Self-supervised learning (SSL) models such as HuBERT and wav2vec 2.0 have significantly advanced speech processing. However, these models incur a substantial computational cost due to the high sampling frequency of speech, which results in extremely long sequences. While subsampling techniques can reduce sequence length and speed up inference, most existing methods employ a fixed compression rate. This approach is suboptimal because different speech tasks have varying requirements: Automatic Speech Recognition (ASR) demands fine-grained temporal resolution, whereas Speaker Identification (SID) or Intent Classification (IC) can operate effectively with much coarser, utterance-level information. Consequently, optimizing for multiple tasks currently requires training multiple specialized models. This paper introduces a "Once-for-All" (OFA) framework that allows a single model to support a continuous range of compression rates, enabling a flexible trade-off between performance and efficiency.
Methods
The OFA framework is built upon DistilHuBERT, a student model trained to distill knowledge from a larger teacher model. The core of the flexible compression mechanism is a Continuous Integrate-and-Fire (CIF) module integrated with a novel $\alpha$ modification strategy.
The framework operates through three key components:
- The $\alpha$ Module: This module predicts weights ($\alpha$) for each input frame. When the cumulative weight reaches a threshold, the model "fires," generating a single output representation for that segment.
- The Once-for-All Parameter ($\lambda$): A scalar parameter $\lambda \in [0, 2)$ is introduced to modify the weights during both training and inference. Values of $\lambda < 1$ result in up-scaling (lower compression) for high-resolution tasks, while $\lambda > 1$ results in down-scaling (higher compression) for efficiency-focused tasks.
- Adaptive Learning: The parameter $\lambda$ can be treated as a learnable variable during downstream task tuning, allowing the model to automatically converge to an optimal task-specific compression rate without manual grid searching.
Evaluation and Results
The model was pre-trained on the LibriSpeech 960-hour dataset and evaluated using the SUPERB benchmark across nine diverse tasks.
Key findings include:
- Task-Dependent Compression Tolerance: ASR performance degrades rapidly with increased compression, while Keyword Spotting (KS) remains resilient even at frame periods approaching one second (~92% accuracy at 960ms).
- Continuous Trade-offs: The OFA approach enables a smooth performance sweep across frame periods from 20ms to 960ms, matching the performance of specialized, fixed-rate models.
- Computational Efficiency: For utterance-level tasks, the model achieved a 90.6% reduction in Multiply-Accumulate operations (MACs) within the Transformer layers at the highest compression levels.
- Effective Adaptive Learning: The adaptive $\lambda$ mechanism successfully identified optimal compression rates for tasks like Phoneme Recognition and Slot Filling, demonstrating its utility for automated task optimization.
Conclusion
The Once-for-All sequence compression framework provides a versatile solution for self-supervised speech models, allowing a single model to serve diverse tasks with varying efficiency requirements. While the current implementation relies on phonetic segments for training guidance, it establishes a robust foundation for flexible SSL models. Future research may extend this "once-for-all" philosophy to other architectural dimensions, such as model depth or width.
Summary updated