Speech recognizers built on Transformers spend most of their compute on attention. Attention compares every audio frame with every other frame, so its cost grows with the square of the number of frames. Much of that work is wasted. Speech has long stretches of silence and steady vowels where neighboring frames are nearly identical. Squeezeformer reports over 95% cosine similarity between adjacent frames by the middle layers. The best existing fix, Adjacent Token Merging (A-ToMe), averages similar neighboring frames layer by layer inside a transducer encoder. A transducer is a recognizer that emits text labels while it steps through audio frames in order. A-ToMe removes 57% of tokens and speeds up GPU inference by 70% with negligible change in word error rate. It has two limits. It only works offline, because each merge decision looks at the next frame, which a live recognizer has not heard yet. And it averages merged frames without weighting, so the encoder forgets how long each merged stretch lasted.
This note proposes a merging rule that removes both limits. Each incoming frame is either folded into the current running segment or starts a new one, based only on frames already seen. Every merged token carries a mass, the count of original frames it stands for. The mass is used in three places: to weight the average, to bias attention so a long segment is not under-counted, and as an input to the decoder that decides when to emit a label. The practical payoff is in streaming recognition. There the attention cost of each new chunk scales with the number of remembered past tokens, so merging lets a model keep minutes of history within the token budget that would otherwise hold seconds. The central claim is testable: at the same average token rate and the same attention-context budget, causal merging that carries mass gives lower word error rate than fixed subsampling or unweighted causal merging. The mass, not the merging alone, is what keeps accuracy intact at aggressive rates.
Core Claim
Progressive token pruning, as developed in vision by DynamicViT and in text by PoWER-BERT, deletes tokens judged unimportant. For speech recognition this is the wrong primitive. CTC and transducer decoders (CTC, connectionist temporal classification, is the other common frame-to-text alignment loss) need an ordered sequence in which every stretch of audio is still represented somewhere. Deleting frames breaks that alignment. Squeezeformer works around the problem by downsampling the middle of the network on a fixed schedule and then upsampling again before the decoder. A-ToMe avoids deletion by merging adjacent frames, but it needs future context and drops duration.
We claim that an online segmentation rule repeated at several layers can deliver the compute savings of progressive pruning to streaming encoders without upsampling, provided each segment's duration travels with it. That rule is causal, adjacent-only, and mass-carrying. The closest prior work is A-ToMe. We differ from it in three ways: decisions are made from the past only; tokens carry mass through attention and into the decoder; and evaluation targets the regime where merging buys long attention memory at a fixed budget, rather than only offline speedup.
Mechanism
At layer $l$, the encoder maintains a current segment with representation $x_s$, mean key $\bar k_s$, and mass $m_s$. When a new token with representation $x_t$, key $k_t$, and mass $m_t$ arrives, it is merged into the segment if two conditions hold. First, $\cos(k_t, \bar k_s) > \tau_l$, where $\tau_l$ is a per-layer threshold. Second, the combined mass stays below a cap $M_l$ that limits how far one token can stretch across a phone boundary. A merge updates the segment as
$$x_s \leftarrow \frac{m_s x_s + m_t x_t}{m_s + m_t}, \qquad m_s \leftarrow m_s + m_t .$$
Otherwise the segment is closed and $x_t$ opens a new one. Comparing keys follows A-ToMe. Comparing against the running segment mean, rather than the single previous frame, is what makes the rule causal and stops a slow drift from chaining many dissimilar frames together.
In later attention layers, the logit from query $i$ to key $j$ receives a bias of $\log m_j$. This is the proportional-attention idea used in bipartite Token Merging for images, so that a token standing for twelve frames gets the attention mass of twelve frames. At the output, $\log m_j$ is appended to the encoder vector fed into the transducer's joint network. The decoder can then learn that a long segment may need several labels, or a long blank, without re-expanding the sequence.
Merging is placed at three or four depths, mirroring the staged schedule in DynamicViT, so early layers still see fine acoustic detail. During training we randomize $\tau_l$ around targets for the token rate. One model can then be run at several speed–accuracy points, which addresses the per-task schedule retuning that limits PoWER-BERT.
Streaming adds one structural detail. Audio arrives in chunks, and each chunk attends to a cache of past tokens capped at $C$ entries. Because cached tokens are merged tokens, a fixed $C$ covers a variable and usually much longer stretch of time. Ragged per-chunk lengths are padded within the chunk only, which bounds the hardware-irregularity penalty the pruning literature warns about. We report measured latency on GPU and on a single CPU core, not just FLOPs, because the scoring and indexing overhead can erase theoretical gains on short inputs.
Research Program
The work has three parts. First, we build the merging module into a standard chunked streaming Conformer transducer and an offline counterpart, both trained on LibriSpeech 960h. Second, we run controlled comparisons that hold fixed either the average token rate in deep layers or the attention-context budget $C$, so that any gain cannot come from simply using more compute. Third, we analyze where merges happen. We test whether merge counts track silence and steady vowels and avoid consonant transitions, using forced alignments. This also bears on the tension the file documents between A-PACK, which finds audio tokens dense and fragile, and FastAV and SpeechPrune, which find most audio tokens disposable. Our expectation is that both are right at different time scales. Rapid transitions are fragile, while silence and steady segments are redundant. A rule that compresses by similarity rather than by an external query should compress the second without touching the first.
A secondary question concerns long-form audio. Speech-Mamba reports Transformer word error rate collapsing to about 48% on 45–60 second concatenated LibriSpeech, and to 67.87% at 100 seconds. We will test whether content-adaptive merging, which keeps deep-layer token counts closer to those seen in training, reduces that collapse in offline encoders. This is a hypothesis to check, not a promise. If the collapse comes from positional extrapolation rather than from length itself, merging will not help, and we will report that.
Baselines
The naive reference point is the same streaming Conformer transducer with no merging, at its usual frame rate and cache size $C$. It sets the accuracy ceiling and the cost floor that every other system is measured against. The file has no reported streaming LibriSpeech result for this configuration, so we will train it ourselves under the same recipe as every other system.
The strongest simple competitor is fixed extra subsampling: one more strided convolution or average-pooling stage that halves or thirds the frame rate, tuned to match our average deep-layer token rate. Its offline analogue in Towards Audio Token Compression in Large Audio Language Models is the reference. There, uniform pooling at 2x gives average WER 6.5 against an unpruned 5.9, uniform sampling gives 7.2, and adaptive segmentation at 3x gives 7.2. Beating fixed subsampling means lower WER at a matched measured latency and matched token rate, at both 2x and 3x.
The strongest existing method is A-ToMe. Its paper reports 57% token reduction, 70% GPU speedup, and 35% single-core CPU speedup with negligible WER impact. No absolute WER value for it is reported on Lacuna. We run it in the offline setting with its published placement between attention and feed-forward blocks, under the same training data, epochs, and tuning budget for thresholds. Offline A-ToMe sees future frames, so it serves as an upper reference. Our goal there is to come within a small margin of it, not to beat it. For offline comparisons we also include Squeezeformer, whose medium model reports 6.50% WER on test-other at 40% fewer FLOPs than a comparable Conformer for its small-medium variant. Being competitive with Squeezeformer at equal FLOPs without its upsampling stage would support the no-upsampling claim.
Two ablations isolate the key ingredient. The first is a naive causal A-ToMe that merges each frame with the previous frame using unweighted averaging, with no mass. The second keeps mass in the average but removes the attention bias and the decoder feature. If the full method does not clearly beat both at aggressive rates (3x and beyond), the claim that mass matters is false.
Why not just…?
Why not just subsample harder with a strided convolution? Uniform reduction ignores how much is happening in the audio. The evidence in the file points against it: in Towards Audio Token Compression uniform methods fall behind boundary-based segmentation at 3x, and A-ToMe's authors report that adjacent merging beats convolutional subsampling at equal reduction. That evidence is offline, however, and streaming may differ. The matched-rate comparison above settles the question directly. If fixed subsampling ties us in streaming, the adaptive rule is not worth its complexity, and we will say so.
Why not just shrink the attention window or use chunked attention with limited memory? This is a real threat. Limited-horizon chunked attention, as used by Mahadeokar et al., already makes cost linear and transcribes audio ten times longer than training length. Our answer is not to replace it but to sit inside it. At a fixed cache size $C$, merged tokens cover more seconds of history. The decisive experiment sweeps $C$ and plots WER against measured per-chunk latency for windowed attention with and without merging. If the two curves coincide, merging adds nothing over windowing.
Why not just use a Mamba hybrid? State space models are recurrent layers with linear cost in sequence length, and Mamba is a popular one. Speech-Mamba shows such a hybrid avoids long-context collapse, reporting 7.71% WER on test-clean-L. However, its authors keep attention blocks to preserve fine acoustic detail, so the quadratic term remains in those blocks. Merging is complementary rather than competing. We include Speech-Mamba's reported number as the long-form reference, and treat merging placed in front of its attention blocks as a follow-up rather than a core claim.
Why not just give A-ToMe a small lookahead? Many streaming systems tolerate 100–300 ms of lookahead, and A-ToMe with one future frame might be enough. We will run A-ToMe with matched lookahead and report emission latency (time from a word's end to its output) alongside WER. Our method only wins this comparison if it reaches equal accuracy at strictly lower emission latency.
Proposed Experiments
Training uses LibriSpeech 960h. Evaluation uses test-clean and test-other, plus the concatenated long-form sets test-clean-L and dev-clean-L at 45–100 seconds used by Speech-Mamba. We report:
- word error rate;
- average tokens per second at each merging depth;
- encoder FLOPs;
- measured real-time factor on an A100-class GPU and a single CPU core;
- emission latency for streaming models.
The first decisive check is the matched-rate streaming comparison against fixed subsampling at 2x and 3x average reduction. The second is the mass ablation at 3x and 4x, where we expect the gap to open, because unweighted merging should increasingly confuse long and short segments. The third is the context-budget sweep against windowed attention described above. As a fourth check, we test transfer without retuning. We evaluate the randomized-threshold model at several rates on test-other without retraining, and to probe beyond read speech, on CSJ-500h spontaneous Japanese, where Deep Sparse Conformer supplies a sparse-attention reference. If the method holds across these, causal mass-carrying merging is a practical way to bring progressive pruning into streaming speech. If it fails the first or third check, the honest conclusion is that fixed subsampling or windowing already captures the available redundancy.
References
- Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
- Accelerating Transducers through Adjacent Token Merging
- DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
- PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
- Token Merging: Your ViT But Faster
- Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs
- FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference
- SpeechPrune: Context-aware Token Pruning for Speech Information Retrieval
- Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models
- Towards Audio Token Compression in Large Audio Language Models
- Deep Sparse Conformer for Speech Recognition