On Compressing Sequences for Self-Supervised Speech Models

Yen Meng, Hsuan-Jui Chen, Jiatong Shi, Shinji Watanabe, Paola García, Hung-yi Lee, Hao Tang

Abstract

Self-supervised learning (SSL) has significantly advanced speech processing through models like HuBERT and Wav2Vec 2.0. However, these architectures are computationally intensive due to their high-resolution processing of audio signals. This paper investigates sequence compression—reducing the number of frames—as a means to improve efficiency. By exploring fixed and variable-length subsampling techniques, we demonstrate that speech models can maintain high performance at significantly lower frame rates, provided the compression strategy aligns with the phonetic structure of the audio.

Introduction

While most research into compressing self-supervised speech models focuses on reducing parameter counts (model compression), the computational burden of Transformers is primarily driven by sequence length. Because Transformer complexity scales quadratically ($O(T^2)$) with the number of frames, processing long audio files at standard resolutions (e.g., 50 Hz) incurs a heavy "computational tax." This study explores whether reducing the frame rate—sequence compression—can yield leaner, faster models without sacrificing downstream task performance.

Methodology

The research utilizes DistilHuBERT as a baseline and evaluates two primary compression strategies:

1. Fixed-Length Subsampling

This approach employs simple downsampling techniques, such as convolution or average pooling with a stride. For example, taking the average of every four frames reduces the sequence length by 75%, but treats all segments of speech as equally informative regardless of content.

2. Variable-Length Subsampling

To address the non-uniform nature of speech—where some phonetic units are longer than others—the authors implemented a Continuous Integrate-and-Fire (CIF) module. The CIF module learns to identify acoustic boundaries (e.g., phoneme transitions) and pools frames within those boundaries into a single representation. To optimize this process, several guidance signals were tested:

  • Cardinality Guidance: Constraining the model to a target number of segments.
  • Segmentation Guidance: Utilizing external signals like smoothed HuBERT codes, unsupervised ASR (wav2vec-U 2.0), or forced alignments to define boundaries.

Experimental Setup

Models were pre-trained on the LibriSpeech 960-hour dataset and evaluated using the SUPERB benchmark. This benchmark covers a diverse range of tasks, including Phone Recognition (PR), Automatic Speech Recognition (ASR), Keyword Spotting (KS), Intent Classification (IC), and Emotion Recognition (ER). Additionally, an ASR-5k task was used to evaluate performance on coarser subword units.

Key Findings

Task-Specific Sensitivity

The utility of high-resolution audio varies by task. Phonetic tasks like Phone Recognition are highly sensitive to compression; performance degrades significantly below 12.5 Hz. Conversely, semantic tasks and ASR using word pieces are remarkably robust, maintaining high accuracy even at frame rates as low as 6.25 Hz.

The Advantage of Variable-Length Compression

At low frame rates (10–12 Hz), variable-length subsampling consistently outperforms fixed-length methods. By allocating more frames to complex, fast-changing sounds and fewer to steady-state vowels or silence, the model preserves more relevant information.

The 10 Hz "Magic Number"

With accurate phonetic boundaries (provided via forced alignment), the researchers found that a model operating at just 10 Hz—a 5x reduction from the 50 Hz baseline—could achieve zero performance degradation. In Phone Recognition, this "topline" model actually outperformed the 50 Hz baseline, achieving a 12.33% Phone Error Rate (PER) compared to the baseline's 16.27%.

Computational Efficiency

Sequence compression leads to dramatic reductions in computational overhead. By subsampling the input to the Transformer layers, the researchers reduced Multiply-Accumulate operations (MACs) in those layers from 207.1G to as low as 27.7G—an 86% reduction. This translates directly into faster inference speeds and lower memory requirements.

Conclusion and Future Work

This study demonstrates that sequence compression is a viable and highly effective strategy for optimizing SSL speech models. While the CIF module currently benefits from external guidance to learn optimal boundaries, future work may focus on fully unsupervised boundary detection. Furthermore, the success of 10 Hz models suggests that even more extreme compression—potentially encoding entire utterances into single vectors—is a promising frontier for efficient speech AI.

Summary updated

BibTeX
@inproceedings{meng2022compressing,
  author = {Yen Meng and Hsuan-Jui Chen and Jiatong Shi and Shinji Watanabe and Paola García and Hung-yi Lee and Hao Tang},
  title = {{On Compressing Sequences for Self-Supervised Speech Models}},
  booktitle = {SLT},
  year = {2022},
  doi = {10.1109/slt54892.2023.10022991},
}
Figure