Speech recognizers that mix convolution with attention, such as the Conformer, spend most of their cost on self-attention. Self-attention lets every audio frame compare itself with every other frame, and its memory grows with the square of the recording length. For long recordings this cost dominates everything else. We propose to test a specific bet: once a cheap depthwise convolution handles local acoustic detail, the global part of the encoder needs only a handful of summary vectors per layer. A depthwise convolution filters each feature channel separately over a short time window and then mixes channels with a 1×1 projection. The summary vectors are built by content-based pooling over time. Because they can be computed as running sums, they make the encoder linear in length and exactly streamable. If the bet holds, full attention, local-window attention, and heavy recurrent state-space blocks are all more machinery than speech encoders need.
The claim can be tested through a single dial: the number of summary slots $k$ per layer. At $k=1$ the global path reduces to ContextNet's squeeze-and-excitation vector, which is one mean vector rescaling channels. As $k$ grows, the global path moves toward attention. Sweeping $k$ against depth, sequence length, and causality measures how much global capacity speech actually uses. That is the main scientific output, and the architecture follows from the answer.
Core Claim
Existing hybrids each provide some global context, but none isolates how much is needed. ContextNet shows that a fully depthwise-separable encoder with one global mean vector per block reaches 2.1%/4.6% LibriSpeech WER without a language model. That result suggests a coarse summary already recovers much of what attention provides. However, the summary is non-causal and position-blind. Squeezeformer reports that adjacent frames in middle Conformer layers have cosine similarity above 95%, which motivates our low-capacity hypothesis: if frames are that redundant, the global signal should be low-rank. Branchformer finds that learned branch weights favor local processing early and late and global processing in the middle. This motivates letting $k$ vary with depth instead of fixing it.
The closest existing system is the Fast Conformer in Investigating End-to-End ASR Architectures for Long Form Audio Transcription. It combines limited-context attention, where each frame attends to a fixed neighborhood, with a few global tokens that attend to and are attended by all frames. It reaches 13.84% WER on Earnings-21 and processes over 8 hours of audio on a 48GB GPU. Our design differs from it in two ways. First, we remove local attention entirely and claim the depthwise convolution makes it redundant. Second, our global summaries have an exact causal form, whereas global tokens that see the whole sequence cannot stream. The ablation that decides the first difference is direct: Fast Conformer with and without its local attention, at matched parameters.
The global operator is simple. At layer $\ell$, $k$ learned query vectors $q_1,\dots,q_k$ each produce a softmax distribution over frames. These distributions pool the frame features $x_t$ into slot vectors $s_j$. Each frame then reads the slots back through a softmax over the $k$ slots:
$$s_j = \sum_t \frac{\exp(q_j^\top W x_t)}{\sum_{t'} \exp(q_j^\top W x_{t'})}\, V x_t, \qquad y_t = \sum_j \mathrm{softmax}_j\!\left(x_t^\top U s_j\right) s_j,$$
Here $W$, $V$, and $U$ are learned projections. The cost is $O(Tkd)$ for $T$ frames and width $d$. In the causal variant, each sum over $t$ runs only up to the current frame. A running numerator and a running log-sum-exp denominator then make streaming inference exact, with constant memory per step.
For $k=1$, with uniform scores, the operator reduces to ContextNet's pooling. For $k=1$ with learned scores, it resembles the global-context-vector attention of MobileViTv2 and SwiftFormer. Neither of those papers was evaluated on audio, and both are flagged as potentially limited by their single vector. So we reuse their pooling technique and test exactly the limitation they leave open.
Research Program
The encoder keeps the simplified Squeezeformer block pattern: a single feed-forward network after each mixing module, with Swish activations throughout. The attention module is replaced by the slot-pooling operator, and the depthwise convolution module is kept. We first fix $k$ uniformly across layers and sweep it over 1, 2, 4, 8, 16, and 64. We then learn a depth-dependent allocation by training with a large slot pool and a sparsity penalty on slot read-back weights, and prune the slots that go unused.
Alongside accuracy, we record diagnostics:
- how sharply each slot's pooling distribution concentrates in time, measured by its entropy;
- whether slots specialize, for example to speaker, channel, or topic segments;
- how slot usage changes with depth, compared against Branchformer's reported pattern;
- how the WER gap to full attention changes as recordings get longer.
The hypothesis predicts several things. The WER-versus-$k$ curve should saturate at small $k$ on LibriSpeech. Saturation should come later on long-form Earnings-21. Middle layers should claim most of the slots. A falsifying outcome would be a curve that keeps improving up to $k=64$ and still trails full attention. That would mean speech needs genuinely position-specific long-range interactions, and the low-capacity view would be wrong.
Baselines
The naive reference point is the same encoder with the global path removed: depthwise convolution and feed-forward blocks only. This is roughly the QuartzNet-style family that the long-form paper shows scales past 13 hours. Every global mechanism must beat it. The $k=1$ uniform-pooling variant reproduces ContextNet-style squeeze-and-excitation inside our block. It is the most important ablation because it isolates whether multiple, content-selected slots matter at all. ContextNet(S) reports 2.9%/7.0% at 10M parameters as an external anchor.
The strongest short-utterance competitor is a Conformer at matched size. Conformer reports 2.7%/6.3% for its 10M-parameter model, but that result uses its own decoder and training recipe. We therefore retrain a Conformer-CTC and a Squeezeformer under our recipe. Squeezeformer-M reports 6.50% test-other WER with CTC, which makes it the most directly comparable published CTC number.
The strongest long-form competitor is the Fast Conformer with limited-context attention and global tokens. Its 13.84% Earnings-21 figure comes from training on 25,000 hours. We will not compare our 960-hour models against that number. Instead, we retrain the Fast Conformer on LibriSpeech 960h with our tokenizer, CTC head, step budget, and hyperparameter search budget, which is equal trials per model. We also retrain it with its local attention removed.
The linear-time recurrent competitor is ConMamba from Speech Slytherin, which reports 3.9% test-clean against 4.3% for a similarly sized Conformer. For streaming, the reference is Centaurus from Let SSMs be ConvNets, with 4.4%/10.2% streaming WER on LibriSpeech.
All models are trained at two sizes, about 10M and about 40M parameters, for the same number of steps on the same hardware. The proposal counts as winning under two conditions. On short utterances, it must come within 0.1 absolute WER of matched full attention. On long-form audio, it must beat the retrained Fast Conformer on Earnings-21 WER at equal or lower memory and with an exact streaming mode. Matching ConMamba's WER at higher throughput would count as a tie on accuracy and a win on simplicity, not a clear victory.
Why not just…?
Why not just use the Fast Conformer? It already provides linear cost and global tokens, so it is the main threat. The proposal differs only if two experiments come out in our favor. The first is the local-attention ablation: if removing local attention costs real WER, our claim that depthwise convolution makes it redundant is wrong. The second is streaming. Global tokens that see the whole sequence are non-causal, and we will test whether a causal Fast Conformer variant degrades more than causal slot pooling does. If both answers favor the Fast Conformer, we will report that.
Why not just use squeeze-and-excitation, as ContextNet does? ContextNet's 2.1%/4.6% suggests one summary vector goes a long way, so this alternative may be nearly sufficient on short read speech. The $k=1$ ablation answers the question directly. Our prediction is that the gap between $k=1$ and small $k$ is small on LibriSpeech and widens on 45–90-minute Earnings-21 calls. A single mean over such calls mixes many speakers and topics. If the gap stays negligible even there, the honest conclusion is that a causal ContextNet is all one needs, and that would be a useful finding in itself.
Why not just use a state-space model such as Mamba? ConMamba already offers linear time and a depthwise convolution module, and our causal operator is a form of linear recurrence. The distinction is that Mamba's state is causal by construction. Speech Slytherin reports that Mamba lags on tasks needing bidirectional context. Our operator has both an exact non-causal form and an exact causal form with the same parameters, so one model can serve offline and streaming use. We will compare WER, throughput, and memory at matched FLOPs in both modes. If ConMamba wins both, slot pooling has no advantage beyond interpretability.
Proposed Experiments
We train on LibriSpeech 960h with CTC and evaluate on test-clean and test-other. For long-form evaluation, we transcribe full Earnings-21 files in a single pass without segmentation. While doing so, we record peak GPU memory and throughput as functions of audio duration up to the point where each model runs out of memory. For streaming evaluation, we use the causal variants on LibriSpeech and compare against the Centaurus figure as an anchor and against retrained causal baselines as the real comparison. We train small models on Google Speech Commands to check whether removing full attention avoids the Conformer divergence reported by Branchformer.
The decisive checks are four. First, the shape of the WER-versus-$k$ curve on short versus long audio. Second, the Fast Conformer local-attention ablation. Third, the $k=1$ versus small-$k$ gap on Earnings-21. Fourth, the per-layer slot allocation compared against Branchformer's local, then global, then local pattern. Together these determine whether a few pooled summaries plus depthwise convolution can replace attention in speech encoders, or whether position-specific long-range interactions are genuinely needed.
References
- ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
- Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
- Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding
- Investigating End-to-End ASR Architectures for Long Form Audio Transcription
- Separable Self-attention for Mobile Vision Transformers
- SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications
- Conformer: Convolution-augmented Transformer for Speech Recognition
- Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis
- Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractions