Linear attention replaces the exact attention weights of a Transformer with an estimate built from random features. A random feature map is a randomized projection whose inner products approximate a kernel, here the softmax similarity between a query and a key. The fewer features you draw, the noisier the estimate. When the features are redrawn at every training step, as Performer-style models do, that noise is multiplicative and zero-centered, much like the noise dropout adds when it randomly zeroes attention weights. Speech models built on the Conformer, which pairs a local convolution module with a global self-attention module, currently set the feature count $M$ and the dropout rate $p$ as two independent knobs. They are usually fixed by habit or tuned one at a time.
We propose that for the attention module these are one knob. Validation error should depend on $(M, p)$ mainly through a single measurable quantity: the total relative noise injected into the attention weights. The convolution module receives no random-feature noise, so its best dropout rate should not move with $M$. If this holds, a practitioner can measure the random-feature noise of a trained model, add only as much dropout as the noise budget still needs, and replace a two-dimensional search with a one-dimensional one. It would also explain why local and global branches want different dropout rates, and it would give a principled reason to redraw features during training even when inference uses a fixed draw.
Core Claim
Let $a_i$ be the exact attention row for query $i$ and $\hat a_i$ its random-feature estimate with $M$ positive random features. Define the relative random-feature noise as $\sigma^2_{RF}(M) = \mathbb{E}\|\hat a_i - a_i\|^2 / \|a_i\|^2$, averaged over queries in a layer. It scales roughly as $c/M$ from the Monte Carlo rate, with a constant $c$ that depends on query and key norms and can be measured directly. Inverted dropout on attention weights at rate $p$ adds independent relative noise $p/(1-p)$, which is the variance of a rescaled Bernoulli mask. The claim is that, for a redraw-at-every-step Conformer, the validation error is approximately a function of
$$\tau = \sigma^2_{RF}(M) + \frac{p}{1-p}$$
plus a separate cost term that depends on $M$ only through inference-time approximation quality. The claim is falsified in either of two cases. The first is that the low-error region of a dense $(M,p)$ sweep does not align with the iso-$\tau$ curves. The second is that removing redraw, which makes the features a fixed draw so that the random-feature noise acts as bias rather than per-step regularizing noise, leaves the optimal $p$ unchanged across $M$.
Where This Sits
Three lines of work motivate the gap. Learning with SGD and Random Features proves, for shallow least-squares models, that $M$ together with step size, iteration count and batch size acts as an implicit regularizer, so explicit ridge penalties become unnecessary. We borrow the intuition that $M$ regularizes. We move it to deep attention, to a sequence loss the theory does not cover, and to the specific interaction with dropout, which that paper does not treat. Recovering loss to followup information using denoising autoencoders found, in shallow tabular autoencoders, that representation width and dropout rate must be tuned jointly. We turn that empirical co-dependence into a quantitative prediction of its shape. Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks showed that architecture rankings flip under proper tuning and defined sensitivity metrics. We reuse those metrics to test whether collapsing two knobs into one actually flattens the landscape.
Other work supplies the method. Rethinking Attention with Performers builds softmax attention from positive orthogonal random features, and Random Feature Attention does the same with random Fourier features. We use the positive-feature estimator from Performers and Orthogonal Random Features because orthogonality lowers $\sigma^2_{RF}$ at fixed $M$. That changes the exchange rate between $M$ and $p$, and so gives us a second, independent test of the theory. The architecture comes from Conformer: Convolution-augmented Transformer for Speech Recognition, which applies one heuristic dropout rate throughout the block.
The closest prior work constrains what counts as new. Efficient conformer-based speech recognition with linear attention (LAC) already linearizes Conformer attention, though with a deterministic normalization kernel and no random features. Its attention therefore has no per-step sampling noise, so our mechanism is absent from it by construction. LAC also changes positional encoding and factorizes the feed-forward layers, which confounds its accuracy gap. Degrees of Freedom for Linear Attention allocates a different $M$ to each layer of language models from Gram-matrix degrees of freedom and treats $M$ purely as a capacity budget. We treat $M$ during training as a noise source and ask what dropout each layer then needs. The two views are compatible, and we test them together. FD-Loss replaces dropout with a deterministic decorrelation loss. That is a different response to dropout's variance and not a competitor on this question.
Research Program
We build a Conformer encoder with a connectionist temporal classification (CTC) output, at roughly the scale of Gulati et al.'s small model. It keeps relative positional encoding in the convolution-free parts where possible and swaps only the attention for positive orthogonal random-feature attention. Everything else stays identical to the exact-attention Conformer, so any gap is attributable to attention alone. That separates us from LAC, which changed several components at once.
The first step is measurement. For each layer we compute $\sigma^2_{RF}(M)$ on real utterances by comparing $\hat a_i$ against exact attention on held-out audio, across $M \in \{16, 32, 64, 128, 256\}$ and for orthogonal versus independent Gaussian features. This gives a per-layer $c$ and tests the $1/M$ scaling on speech queries and keys.
The second step is a dense grid over $M$ and attention-dropout $p$, with convolution-module dropout $p_{conv}$ swept separately. We run it under three regimes: redraw every step, a fixed single draw, and orthogonal versus unstructured features. Sweeps use a 100-hour subset of LibriSpeech and AISHELL-1. We then fit error as a function of $\tau$ alone versus a free function of $(M,p)$ and report how much variance the one-dimensional model explains.
The third step turns the theory into a procedure. We choose a target $\tau$ by a one-dimensional search, and for any $M$ we set per-layer $p$ to top up the measured $\sigma^2_{RF}$ to that target. When combined with degrees-of-freedom allocation, layers given small $M$ automatically receive little or no dropout.
To guard against overtuning the selection, as warned in Overtuning in Hyperparameter Optimization, all selection uses dev sets only. Test sets are scored once per final configuration, and we report seed variance.
Baselines
The naive reference is the random-feature Conformer at a single default $M$ (equal to head dimension), with the Conformer's uniform default dropout and no search. It shows what a practitioner gets today.
The strongest existing reference for accuracy is exact-attention Conformer small. Gulati et al. report 2.7% / 6.3% WER on LibriSpeech test-clean / test-other for the 10M model (Conformer). We retrain it under our exact recipe and tuning budget rather than comparing against the published number, since our sweeps use less data.
The strongest existing efficient competitor is LAC. It reports 5.8% WER on LibriSpeech test-other and 5.02% CER on AISHELL-1, against 4.88% for its full Conformer (Efficient conformer-based speech recognition with linear attention). We reimplement its normalized linear attention inside our otherwise identical encoder and give it the same trial budget over learning rate, $p$ and $p_{conv}$. For the 100-hour subset there is no reported result on Lacuna for any of these systems, so all subset numbers are ours.
The key comparison is between the noise-matched one-dimensional search and a joint random or Bayesian search over $(M, p, p_{conv}, \text{learning rate})$ with the same number of full training runs. Beating it means one of two outcomes at matched trials. Either we reach equal or lower dev error, or we reach the joint search's final error with at most half the trials.
Two ablations isolate the ingredient. Fixed-draw features without redraw remove the per-step noise while keeping the kernel approximation, and the theory predicts the $M$–$p$ coupling disappears there. Using unstructured instead of orthogonal features changes $c$, and the theory predicts the optimal $p$ shifts by the measured change in $\sigma^2_{RF}$.
Why not just…?
Why not just run a standard joint hyperparameter search over $M$ and $p$? This is the real threat. Given enough trials, a Bayesian optimizer will find the ridge whether or not anyone understands it. Our answer is empirical and can fail: the matched-budget comparison above measures whether knowing the ridge's shape in advance saves trials. If joint search reaches equal error just as fast, the practical contribution collapses. What would remain is a measured explanation of why the optimum moves, which also predicts how it transfers to new $M$ or feature types without re-searching. The transfer test is part of the experiments.
Why not just drop attention dropout and let $M$ regularize, as the SGD random-feature theory suggests? Learning with SGD and Random Features supports that ridge penalties are redundant in shallow least squares, and if that carries over, $p=0$ is optimal everywhere. Our theory predicts something narrower. $p=0$ should be optimal only where $\sigma^2_{RF}(M)$ already exceeds the target $\tau$, that is, at small $M$. At large $M$, dropout should still help. The $(M,p)$ grid settles this directly.
Why not just use exact attention with an IO-aware kernel, since LibriSpeech utterances are short? For short utterances this is a fair objection to the efficiency case, and we do not claim speed gains there. The claim concerns how regularization should be set once random-feature attention is used, which matters for the long-form audio where quadratic cost binds. We therefore also evaluate on concatenated multi-utterance LibriSpeech segments. On those we report dev WER and wall-clock time for both, and we state plainly if exact attention remains preferable at our lengths.
Why not just allocate $M$ per layer by degrees of freedom and keep default dropout? That is the natural extension of Degrees of Freedom for Linear Attention to speech, and we run it as a baseline. Our prediction is that it helps more when paired with noise-matched dropout. Low-$M$ layers under default dropout receive noise on top of noise, and the per-layer grid shows whether that over-regularizes them.
Decisive Checks
Three results would settle the claim. First, on both LibriSpeech-100h and AISHELL-1, dev error should collapse onto a single curve in $\tau$ under redraw, and that collapse should vanish under fixed draw. Second, the optimal $p_{conv}$ should stay flat across $M$ while the optimal attention $p$ follows the iso-$\tau$ prediction, which would confirm that the local and global branches need different regularization for a mechanistic reason. Third, if random-feature noise acts as regularization rather than a capacity limit, a model trained with small $M$ and redraw, then evaluated with a larger fixed $M$, should improve the way dropout-trained networks improve at inference. If instead $M$ acts only as capacity, evaluation-time $M$ will not matter.
We also report landscape flatness using the percentage of best-equivalent runs and Rel@k from Wiemerslage et al., computed over $\tau$ versus over raw $(M,p)$. Finally, we carry the best subset configurations to full LibriSpeech for the exact Conformer, LAC and our model at equal parameter count and tuning budget.
References
- Learning with SGD and Random Features
- Recovering loss to followup information using denoising autoencoders
- Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks
- Rethinking Attention with Performers
- Random Feature Attention
- Orthogonal Random Features
- Conformer: Convolution-augmented Transformer for Speech Recognition
- Efficient conformer-based speech recognition with linear attention
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency
- FD-Loss: Supervised Feature Decorrelation as a Scale-Invariant Replacement for Random Dropout
- Overtuning in Hyperparameter Optimization