Temporal Frame Stacking for Efficient Speech Transformers

29 concepts22 papers1 proposal

Audio and speech processing models face a fundamental computational challenge: the sheer length of their input sequences. While a text-based Transformer might process a few hundred words in a sentence, a speech model processing the corresponding audio must handle thousands of individual frames. Traditionally, acoustic features are extracted at high temporal resolutions—such as 2ms or 10ms intervals—to ensure no fine-grained acoustic detail is lost.

However, this high resolution introduces severe computational bottlenecks. In Transformer architectures, the self-attention mechanism scales quadratically ($O(T^2)$) with the sequence length $T$, leading to massive memory footprints and slow training times (Wanqiu Lin et al.; Andros Tjandra et al.). For autoregressive models, such as decoder-only large language models (LLMs) or neural audio codecs, long sequences translate directly to an excessive number of sequential decoding steps, severely limiting inference throughput (Ankit Gupta et al.; Shih-Lun Wu et al.).

To solve this, a prominent research direction focuses on temporal frame stacking and downsampling. By aggressively reducing the sequence length early in the network architecture, researchers across the field have unlocked massive speedups and memory savings with minimal impact on accuracy.

The Core Intuition: Exploiting Acoustic Redundancy

Why can we afford to throw away temporal resolution without destroying the model's performance? The answer lies in the continuous nature of speech and audio signals.

Unlike text, where every token represents a discrete semantic unit, audio is highly continuous and locally redundant. A single phoneme or musical note typically spans tens or hundreds of milliseconds. Consequently, consecutive 10ms frames capture nearly identical acoustic properties.

When a model processes raw, high-resolution frames, the early layers waste significant computation performing self-attention over highly correlated, neighboring vectors. By stacking or downsampling these frames early in the pipeline, we compress this local redundancy. This forces the attention mechanism to operate on a macro-level, focusing on sparser, more distinct acoustic transitions rather than redundant local noise (Wanqiu Lin et al.).

Mechanisms of Temporal Compression

Researchers have developed several distinct strategies to compress the temporal dimension of audio sequences, depending on where the compression occurs in the pipeline.

1. Frame Stacking and Concatenation

The most straightforward approach is to concatenate $N$ consecutive feature frames into a single, higher-dimensional vector, effectively reducing the sequence length by a factor of $N$.

High-Resolution Frames:  [f1] [f2] [f3] [f4] [f5] [f6]
                            \   /     \   /     \   /
Stacked Representation:    [ f1+f2 ] [ f3+f4 ] [ f5+f6 ]

Andros Tjandra et al. utilize a time-stacking layer early in a self-supervised pipeline to mitigate the quadratic complexity of Transformers for language identification. Similarly, Georgios Paraskevopoulos et al. and Quan Pham et al. demonstrate that grouping consecutive frames reduces memory overhead and stabilizes the training of exceptionally deep self-attention networks by presenting them with shorter, more stable sequences.

2. Subsequence Pooling and Averaging

Instead of concatenating frames (which increases the input dimension), some architectures compress the sequence length by averaging consecutive vectors. This is particularly common when interfacing speech encoders with text-based LLMs. For example, in music captioning, Zihao Deng et al. average subsequences of temporal representations to ensure that long audio clips can fit comfortably within the fixed context window of a text transformer.

3. Learned 1D Convolutions and Strides

Rather than using fixed mathematical operations like stacking or averaging, models can learn how to downsample. Salah Zaiem et al. explore input sequence downsampling via learned 1D convolutions and signal decimation, showing that reducing the frame rate by a factor of three can slash inference time by over 60% while maintaining competitive word error rates.

4. Modifying the Front-End Frame Size

An alternative to downsampling an already-extracted feature sequence is to change the resolution at the source. Danilo de Oliveira et al. demonstrate that by simply increasing the initial Short-Time Fourier Transform (STFT) frame size from 2ms to 32ms, a speech enhancement Transformer achieves an eightfold reduction in computational operations and a sixfold increase in execution speed. Similarly, Shucong Zhang et al. show that lowering the output frequency of the acoustic feature extractor itself drastically cuts down both memory consumption and total training time.

Impact Across Different Architectures

The benefits of temporal frame stacking manifest differently depending on the underlying model architecture, but the end goal remains the same: breaking the sequence-length bottleneck.

Transducers and Mobile Deployment

For streaming speech recognition on edge devices, processing speed and memory constraints are paramount. Yanzhang He et al. integrate a time-reduction layer early in an RNN-Transducer (RNN-T) encoder, accelerating processing by nearly 1.7 times for on-device deployment. To minimize the frequent communication overhead between acoustic and language models during decoding, Jinhwan Park et al. adopt sub-word units that allow the model to operate at a significantly lower frame rate.

Taking a different approach to frame reduction, Hainan Xu et al. introduce "multi-blank" transducers. Instead of compressing all frames uniformly, their model uses variable-duration skip symbols to dynamically bypass redundant frames during stretches of silence or stable acoustic signals, yielding up to a 140% speedup during inference.

Large-Scale Speech Models and LLMs

As speech models scale up, aggressive downsampling becomes essential for viability. In reproducing Whisper-style architectures, Yifan Peng et al. demonstrate that downsampling speech input to a 40ms resolution (compared to the standard 20ms) significantly lowers GPU memory consumption and accelerates training without substantial performance loss.

When giving LLMs speech capabilities, Yassir Fathullah et al. stack consecutive audio frames so that substantial durations of audio can be represented within a single context window slot. Similarly, in the realm of neural codec language models, Shih-Lun Wu et al. show that increasing the temporal duration represented by a single token doubles inference speed by reducing the number of autoregressive steps the language model must take.

Word-Level Modeling

The logical extreme of temporal downsampling is to align the acoustic frame rate directly with linguistic units. Ronan Collobert et al. introduce a direct-to-word speech recognition model that utilizes much larger temporal strides. Because words naturally span significantly longer durations than characters or phonemes, the acoustic model can downsample audio aggressively, drastically improving decoding speed and reducing memory usage without sacrificing accuracy.

Key Foundational Work

Purely Sequence-Trained Neural Networks for ASR Based on Lattice-Free MMI (2016) — contributed 1 concept elements to this direction.
Fully Neural Network Based Speech Recognition on Mobile and Embedded Devices (2018) — contributed 1 concept elements to this direction.
Streaming End-to-end Speech Recognition For Mobile Devices (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_frame_stacking_f61d877d,
  title = {temporal frame stacking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-frame-stacking/rdr_f61d877d13a544e18efbc8cf89be9b84},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_frame_stacking_f61d877d_v1,
  title = {temporal frame stacking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-frame-stacking/rdr_f61d877d13a544e18efbc8cf89be9b84/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake