Temporal Frame Stacking Speedups Depend on the Hardware

Speech models see audio as a long sequence of short frames, typically one every 10 to 20 ms. Many systems shrink that sequence early by frame stacking, which glues several consecutive frames into one wider vector or pools them, so every later layer has fewer positions to process. The usual justification is that self-attention cost grows with the square of sequence length, so halving the length should save much more than half the work. This proposal argues that this justification is mostly wrong for realistic speech encoders, and that the actual wall-clock benefit of stacking depends on the device and batch size. On some deployments stacking gives nearly its full proportional speedup. On others, such as a GPU decoding one stream at a time, it gives almost nothing while still costing accuracy. If that holds, a single fixed stacking rate chosen at training time will be wrong for some of the hardware a model is shipped to.

The work has two parts. First, a measured, operator-by-operator account of where stacking saves time on four classes of hardware: a consumer GPU, a datacenter GPU, a laptop system-on-chip and a server CPU. Second, a temporally elastic encoder: one network, fine-tuned once, that can run at several stacking rates chosen at deployment time. The test of whether the elastic model is worth having is simple. At equal measured latency on each device, it must match a set of separately trained fixed-rate models and beat any single fixed-rate model.

Core Claim

For a Transformer layer with model width $D$, feed-forward width $4D$ and $T$ positions, the matrix-multiply work is about $24TD^2$ and the attention-score work is about $4T^2D$. The quadratic part's share of the layer is therefore

$$ s(T) = \frac{T}{T + 6D}, $$

which follows from dividing $4T^2D$ by the sum of the two terms. For a WavLM-Large-sized encoder ($D = 1024$) at 50 frames per second on a 15-second utterance ($T = 750$), $s \approx 0.11$. So stacking by a factor $r$ saves work roughly in proportion to $r$, not $r^2$.

Whether that proportional saving turns into time then depends on whether the layer's matrix multiplies stay large enough to keep the hardware busy. As $B \cdot T / r$ shrinks, where $B$ is batch size, those multiplies move below the "roofline knee." That is the point described in Searching for Fast Model Families on Datacenter Accelerators at which a kernel stops being limited by arithmetic and becomes limited by loading weights and activations from memory. Past that point, latency stops falling with $r$.

This yields three predictions, each specific enough to be wrong:

  1. On CPUs, the knee sits at small token counts, so stacking pays off almost linearly. This matches the 61.34% CPU time reduction at 3x downsampling reported by Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models.
  2. On a datacenter GPU at batch 1 or in chunked streaming, latency is set by weight loading, and stacking beyond about 2x buys little.
  3. The stacking rate that minimizes latency at fixed accuracy therefore differs across devices and batch sizes, and so does the choice between stacking and channel compression.

Relation to Prior Work

Several papers motivate the claim by showing stacking works but never asking why it is fast:

Two papers come closest to joining the two directions:

- On the (In)Efficiency of Acoustic Feature Extractors shows that a feature extractor which looks cheap on paper (Leaf) is slow without tuned GPU kernels, and that halving the frame rate to 25 Hz cut pre-training from 670 to 180 GPU-hours. It studies only the input stage, at one fixed rate.

  • Zaiem et al. measure real latency of downsampling, but only on CPU and only at a fixed 3x rate before the encoder. They also report a sharp loss of accuracy when fine-tuning data falls to about 10 hours.

This proposal differs from both in three ways. It covers stacking placed inside the encoder, it measures across devices and batch sizes, and it replaces the fixed rate with a rate chosen at run time.

The method borrows from two places. The elastic training follows LightHuBERT, a once-for-all network that samples sub-networks of different width and depth during training; its downsampling stage was explicitly left static. The idea that redundancy grows with depth follows the Hourglass Tokenizer, which prunes frames in the middle layers of a video pose Transformer and restores them with cross-attention.

Two papers constrain the design:

  • ShuffleNet V2 shows that reshapes, concatenations and fragmented operations carry real memory cost that operation counts miss.
  • Compute-Optimal Is Not Cluster-Optimal shows that rankings by operation count and by wall-clock time can flip, which is exactly the kind of flip predicted here.

LiteASR, described below, is the main competing approach.

Research Program

Measurement study. Start from WavLM Large with a CTC head on LibriSpeech, the setting of Zaiem et al. Insert a merge step after layer $k$ that reduces positions by $r \in \{1, 2, 3, 4, 6\}$, for $k \in \{0, 4, 8, 12\}$. Three merge types are compared:

  1. concatenation followed by a linear projection;
  2. a strided convolution;
  3. mean pooling.

Latency does not depend on trained weights, so the full grid can be timed with untrained checkpoints. The devices are an RTX 4090, an A100-class GPU, an Apple M1 Pro and a multi-core x86 CPU. Each is tested at batch sizes 1, 8 and 32 and in a chunked-streaming mode. Each run is split into matrix multiplies, the attention kernel (FlashAttention where available), element-wise operations and the merge itself. Every operator is placed on that device's roofline.

The deliverable is a measured curve of latency against $B \cdot T / r$ for each device, showing where the knee falls. A secondary question is whether the merge type matters. ShuffleNet V2's analysis suggests concatenation's wider projection could be memory-costly, but my expectation is that its cost is negligible next to placement and rate.

Elastic stacking model. During fine-tuning, each batch samples a rate $r$ from the set above, with $k$ either fixed or also sampled. Each rate gets its own small merge projection, while all Transformer weights are shared. Shorter sequences give weaker training signal, so in the spirit of LightHuBERT's sandwich-style sampling the full-rate model's outputs serve as a distillation target for the stacked passes. LightHuBERT reports gradient interference between sub-networks, so two signs of trouble are tracked: falling full-rate accuracy, and the gap to separately trained per-rate models. At deployment, the rate is picked per device and batch size from the measured curves, as the largest accuracy-preserving rate that still lands before the knee.

Baselines

Unstacked encoder. WavLM Large fine-tuned at the native rate with no stacking. This is the "do nothing" reference that every speedup and accuracy number is measured against. It uses the same data, the same fine-tuning steps and the same decoding, including the same language model where one is used.

Fixed-rate models, one per rate and placement. These follow Zaiem et al., who report a LibriSpeech 100h test WER going from 3.31 to 4.12 with a language model at 3x downsampling, together with a 61.34% CPU time cut. This is the strongest competitor on accuracy, since each model is trained for exactly one rate. Each gets the same fine-tuning budget as the elastic model. Because they are trained separately, their total training cost is several times higher, and that difference is reported. The elastic model beats this baseline if its WER at each rate is within 0.2 absolute of the matching fixed model while needing one training run. It also beats it if, at equal measured latency on any device, it is better than the single fixed rate a practitioner would ship everywhere.

LiteASR channel compression. LiteASR compresses the Whisper encoder by low-rank factorization of its weights, using custom Triton kernels. It reports a 1.57x peak speedup on an RTX 4090 and 1.4x average on the M1 Pro. It is the strongest alternative way to cut latency without touching sequence length. It is run with its public code on the same devices, and stacking is compared at matched measured latency on ESB and LibriSpeech. Whisper-based comparisons apply stacking inside the Whisper encoder so both methods share a backbone. Stacking wins on a device if it gives lower WER at equal latency there.

Front-end rate reduction. A 25 Hz feature front end as in Parcollet et al. This isolates whether stacking inside the network beats simply cutting the rate at the input.

Ablations. A version without the distillation target, and a version with stacking only at $k = 0$. Together these isolate the two ingredients: rate elasticity and the choice of where in the network to stack.

Why not just…?

Why not just train one fixed-rate model per deployment target? This is the real threat. If the knee analysis shows that the best rate is the same on all four devices, a single fixed model is enough and the elastic model has no reason to exist. The measurement study settles this first and gates the second phase. Even when rates do differ, separate models multiply training and storage costs. The elastic model is judged directly against them at equal accuracy.

Why not just compress channels instead, as LiteASR does? Channel compression also cuts work roughly in proportion and needs custom kernels to become a real speedup. The two methods should behave differently depending on the regime. When matrix multiplies are limited by weight loading, as at batch 1 on a GPU, fewer weights should help more than fewer positions. When they are limited by arithmetic, fewer positions should help as much or more. The matched-latency comparison across devices tests this. If channel compression wins everywhere, that is reported as a result. Combining the two is also measured.

Why not just lengthen the input frames? Longer frames at the input, as in Efficient Transformer-based Speech Enhancement Using Long Frames, or a 25 Hz front end, are the cheapest option and are tested as a baseline. Zaiem et al.'s low-data degradation suggests that reducing the rate at the input throws away detail that later stacking can keep. The $k = 0$ versus $k = 8$ comparison on the 10h LibriSpeech and Buckeye subsets tests this directly.

Why not just batch larger to restore GPU utilization? Larger batches move a workload back above the knee. However, batch size is set by the serving setting, not by the modeler, and streaming and on-device use run at batch 1. That is why batch size is a measured variable rather than a free choice.

Proposed Experiments

The first decisive check tests the core claim. If, for any GPU at batch 1 or in streaming, going from 2x to 4x stacking gives less than a 15% latency reduction while CPU gives near-proportional reduction, the claim holds. If all devices scale proportionally, the claim is false and the elastic model loses its motivation.

The second check compares rankings. The best rate-and-placement configuration should change between the measured Pareto fronts of WER against operation count and of WER against latency on at least two of the four devices.

The third check tests the elastic model on LibriSpeech test-clean and test-other with 100h and 960h fine-tuning, and on ESB for the Whisper comparison. Against the fixed-rate models, it must stay within 0.2 WER absolute at each rate.

For streaming context, results are reported alongside the 80 ms latency figures of Emformer (3.01% test-clean, 7.09% test-other) and ConvT-T (3.5% and 8.3%), as reference points rather than direct competitors. No reported latency breakdown was found on Lacuna for DOTA's 8x versus 12x stacking, so that configuration is re-timed rather than quoted.