Abstract: Pre-trained speech models, such as wav2vec 2.0, have significantly advanced speech-related tasks, including speech recognition and translation. However, their applicability in streaming scenarios is limited because these models are trained on complete utterances, leading to a mismatch with incremental streaming inputs. This paper identifies three critical design aspects within the architecture of wav2vec 2.0 and proposes a novel model, wav2vec-S, which incorporates simple modifications to ensure consistent speech representations during both training and inference phases for streaming speech inputs. Furthermore, we demonstrate that wav2vec-S models can be efficiently adapted from pre-trained wav2vec 2.0 models through continued pre-training and effectively finetuned to meet various latency requirements in downstream applications. Experiments on speech recognition and translation tasks show that wav2vec-S outperforms strong baseline models and achieves a superior balance between quality and latency.
The Challenge of Offline Bias
Pre-trained speech models like wav2vec 2.0 have revolutionized the field, but they suffer from a significant "offline" bias. Typically trained on complete utterances, these models utilize bidirectional self-attention and convolution-based positional encodings that inherently require "future" audio context.
When deployed in streaming scenarios—such as real-time captioning or live translation—two primary issues emerge:
- Architectural Lookahead: Components require audio that has not yet been spoken.
- Statistical Mismatch: Normalization techniques (like Group Normalization) often calculate statistics across the entire sequence. During streaming, the model only sees partial chunks, leading to a discrepancy between training and real-time inference statistics.
Methodology: The wav2vec-S Architecture
The authors propose wav2vec-S, which introduces surgical modifications to the wav2vec 2.0 architecture to eliminate future-context dependencies:
- Feature Normalization: Group Normalization (GN) is replaced with Layer Normalization (LN). Unlike GN, which calculates statistics across the sequence length, LN operates exclusively along the feature dimension, ensuring representation consistency regardless of input length.
- Positional Encodings: The standard 1D convolution (with a 2.5-second kernel) is swapped for absolute sinusoidal positional encodings, providing stable position information without future lookahead.
- Self-Attention: The model implements block-wise unidirectional self-attention, allowing it to attend to all past context while restricting future lookahead to a small, fixed "right-side" context within a processing block.
- Dynamic Latency Training: By randomly varying block sizes during training, the authors create a versatile model capable of deployment at different latency settings (e.g., 160ms to 640ms) without retraining.
Efficient Adaptation via Continued Pre-training
A key contribution of this work is the efficiency of model adaptation. Rather than training from scratch, wav2vec-S is initialized with standard wav2vec 2.0 weights. Continued self-supervised pre-training on unlabeled data "warps" existing knowledge into a streaming-compatible format. This approach reaches the performance of from-scratch training in just 25% of the time (100k steps vs. 400k steps).
Experimental Results
Quality-Latency Trade-offs
In streaming Speech Translation (ST) tasks for English-German (EnDe) and English-Spanish (EnEs), wav2vec-S consistently outperformed strong baselines like CAAT and MU-ST by over 1 BLEU point across nearly all latency regions.
Robustness in Low-Data Scenarios
wav2vec-S demonstrates significant advantages when fine-tuning data is limited. For instance, using only 100 hours of LibriSpeech data, wav2vec-S reduced the Word Error Rate (WER) by approximately 6-10 absolute points compared to models not pre-trained for streaming.
Conclusion and Future Work
wav2vec-S provides a streamlined, effective framework for transforming offline speech models into high-performance streaming encoders. While it achieves state-of-the-art results, challenges remain in capturing long-range syntactic dependencies in languages with different word orders (like German). Future research may explore alternative sequence-independent normalization methods and extend these principles to other pre-trained architectures like HuBERT or WavLM.
Summary updated