Speech encoders built for recognition usually add small sliding filters (convolutions) to a Transformer, on the theory that speech has fine local structure, such as formant transitions and phone boundaries, that attention alone learns slowly. The Conformer is the standard example of this design. A competing view holds that enough data makes such built-in assumptions unnecessary, because a plain Transformer will learn its own local filters. Self-supervised pre-training on unlabeled audio is the setting where this question matters most, since unlabeled hours are cheap and plentiful. No one has measured whether the convolution's advantage shrinks, holds, or grows as unlabeled data increases, or whether the answer depends on the downstream task: recognizing words, identifying speakers, or reading emotion.
We propose to express the value of the convolution in hours of unlabeled audio. For each data scale and each downstream task family, we ask how many extra pre-training hours a pure Transformer needs to match a Conformer trained on a given amount. If this "hours-equivalent" ratio falls toward one as data grows, data substitutes for the inductive bias. If it stays flat or diverges, the bias does work that data does not replace. A third arm replaces the learned convolutions with fixed, non-learned shifts and averages. This arm tests whether the useful part is the learned filters or simply cheap access to neighbouring frames.
Core Claim
The local bias in hybrid speech encoders is a data-dependent quantity, and it can be measured. We hypothesize that its value differs by task family. For content tasks (phoneme recognition, recognition of words), the hours-equivalent advantage of convolutions should decay with pre-training scale. For speaker and paralinguistic tasks, which depend on fine spectral detail held in lower layers, it should persist. The existing evidence pulls in opposite directions. Scaling Laws vs Model Architectures fit scaling curves across more than ten text architectures. They found that convolutional models match Transformers on pre-training loss but lag downstream, and that the vanilla Transformer has the steepest upstream exponent (0.54). This motivates our question, but it has never been tested on continuous audio or on hybrid blocks. OWSM v3.1 found a hybrid (E-Branchformer) beats Transformers at 180k supervised hours. Conformer without Convolutions found learned filters replaceable by fixed shifts and averages on 960 supervised hours. Neither study used self-supervision or varied scale. Our result would reconcile these findings in the regime where data scale is actually cheap.
The prediction is falsifiable in both directions. If the ratio decays to near one for every task family, the convolution is a small-data crutch for self-supervised speech, and large-scale self-supervised encoders should drop it. If the fixed-operation arm tracks the learned Conformer at all scales, the learned filters were never the point.
Research Program
We train three encoder families that differ only in the local module. The first is a plain Transformer. The second is a Conformer with learned depthwise convolutions, following Gulati et al.. The third is a Conformer whose depthwise convolutions are replaced by the fixed 25% average / 75% shift scheme of Conformer without Convolutions. The input subsampling is identical across arms, at a 4× rate giving 40 ms frames. This keeps temporal resolution from becoming a hidden confound. NEST used 8× subsampling to 80 ms frames, and Squeezeformer showed that downsampling choices interact with the local module. We hold both fixed and treat them as out of scope.
The training objective is BEST-RQ. This is masked prediction of discrete targets produced by a frozen random projection and codebook, the objective used by NEST and Polynomial Mixing. We choose it for a methodological reason beyond cost. Its targets do not depend on the model, so pre-training losses are directly comparable across architectures. HuBERT-style targets, re-clustered from each model's own features, would not be. This comparability is what lets us test the upstream–downstream decoupling that Tay et al. reported in text.
The grid spans unlabeled data of 100, 1k, 10k, and 60k hours. These are nested subsets of LibriLight, with VoxPopuli mixed in at the top scale to echo the NEST corpus. Model sizes are roughly 25M, 95M, and 300M parameters. Each architecture is width-matched to equal training FLOPs, not equal parameter count, since convolutions and attention differ in cost per parameter. For each architecture and model size, we fit a data scaling curve to pre-training loss and to each downstream metric $M$:
$$M(D) = E + (A/D)^{\alpha},$$
where $D$ is unlabeled hours, $E$ is an irreducible floor, and $A, \alpha$ are fitted per arm. This is the saturating power-law form used by Pu et al., who fitted the loss-versus-parameters version for Transformer self-supervised models up to 42.5M parameters and 360 hours. The hours-equivalent ratio at scale $D$ is $k(D) = D_T / D$. Here $D_T$ is the data at which the Transformer's fitted curve reaches the Conformer's value at $D$. We report $k(D)$ with bootstrap intervals over three seeds at the smaller scales. To see where any persistent advantage lives, we also run frozen layer-wise probes for speaker identity and phone identity. These test whether the convolution's benefit sits in the lower layers, which earlier work found carry speaker and acoustic detail.
The honest constraint is compute. The 60k-hour, 300M cells in three arms are the expensive corner. If the budget forces a cut, we drop the 300M size at 60k hours rather than a data scale. The question is about data, and four points along $D$ are the minimum needed to separate a change in exponent from a constant offset.
Baselines
The naive reference point is log-mel filterbank features fed directly to the same downstream heads. SUPERB reports that self-supervised representations beat these features on nearly every task. Including this point checks that every arm, including the smallest at 100 hours, actually learned something. Any arm that does not clearly beat filterbanks at 100 hours is undertrained, and its curve is not interpretable.
The main competitor is the plain Transformer trained with the identical BEST-RQ recipe. The reviewer's default belief is that, at enough scale, it matches or beats hybrids. It gets the same data subsets, FLOPs, masking schedule, and a tuning sweep of equal size over learning rate and warm-up. Equal tuning matters because OWSM v3.1 notes that hybrid encoders are sensitive to warm-up schedules, so a Transformer given one setting could lose for the wrong reason. The Conformer "beats" it on a task family at scale $D$ only if the bootstrap interval of $k(D)$ excludes one at the largest scale.
The fixed shift-and-average Conformer is the ablation that isolates the key ingredient. It keeps the Conformer's local wiring but removes learned local filters. The question is whether learned filters justify their cost under self-supervision.
To anchor absolute quality against published systems, we compare our top cells with reported numbers. SUPERB reports 2.94% word error rate on LibriSpeech test-clean for frozen HuBERT Large with a 2-layer BLSTM head. NEST reports 14.76% diarization error rate on DIHARD3 and 89.79% intent accuracy on SLURP for NEST-XL at 100k hours. Polynomial Mixing reports 8.31% word error rate on test-clean for a 95M Conformer pre-trained with BEST-RQ on LibriSpeech-960 and fine-tuned on 100 hours, against 8.59% for attention in their setup. Our 1k-hour, 95M Conformer cell, fine-tuned identically on 100 hours, should land near that 8.59% figure as a sanity check on the recipe. We do not aim to beat these systems. Matching the Polynomial Mixing recipe point is the reproducibility gate.
Why not just…?
Why not just scale a plain Transformer and stop worrying about local bias? That is exactly the hypothesis one of our arms tests, and the evidence for it does not cover this setting. Radford et al. showed robust recognition from a Transformer on 680k weakly labeled hours, but did not compare against a hybrid at matched data. OWSM v3.1 found a hybrid lowered multilingual error from 18.8% to 15.2% at 180k supervised hours. Neither used masked self-supervised pre-training. If $k(D)$ falls to one by 60k hours on all task families, the reviewer is right and we report it as the main finding.
Why not just use fixed shifts and averages instead of convolutions? This is a real threat. Conformer without Convolutions found fixed operations slightly improved word error rate for Conformer, Squeezeformer, and FastConformer, by 0.12, 0.62, and 0.20 points, on supervised LibriSpeech. Masked prediction may need more than fixed local mixing, because it must reconstruct masked spans from context, and speaker tasks may need learned spectral filters. We do not argue this point in advance. The fixed arm runs at every grid cell, and if it tracks the learned Conformer, we recommend it.
Why not just compare existing checkpoints such as HuBERT, WavLM, and NEST? These differ in data, objective, frame rate, and augmentation all at once. NEST alone uses 8× subsampling, multi-speaker mixing, and 100k hours. Any architectural conclusion drawn from them is confounded. Point comparisons at one scale also cannot separate a constant offset from a change in scaling exponent, and that distinction is the whole question.
Why not just add speaker diversity instead of studying architecture? More Speaking or More Speakers? found self-supervised wav2vec 2.0 is volume-hungry: word error rate fell from 49% to 23% going from 100 to 400 unlabeled hours, with little gain from more speakers. This supports scaling hours, as we do. Because our subsets are nested LibriLight cuts, speaker count grows with hours. We log speaker counts per subset, so the small-scale end of our curves can be checked against that plateau.
Proposed Experiments and Decisive Checks
All arms are evaluated frozen on SUPERB using its learned layer-weighted sum and light heads. The tasks cover content (phoneme recognition and word recognition on LibriSpeech), speaker (identification and verification on VoxCeleb1), semantics (intent classification on Fluent Speech Commands), and paralinguistics (emotion on IEMOCAP). We also fine-tune each arm on LibriSpeech 10h and 100h and report word error rate on test-clean and test-other.
The first decisive check is the shape of $k(D)$ per task family. Our hypothesis is supported if the content-task ratio declines across the four data scales while the speaker and emotion ratio stays above one, with intervals that separate them at 60k hours.
The second check is decoupling. If the Conformer and Transformer reach the same BEST-RQ loss at a given $D$ but differ downstream, then pre-training loss cannot be used to choose speech architectures. That would be the speech analogue of the text finding of Tay et al..
The third check uses the layer probes. They test whether any persistent advantage concentrates in the lower layers, which would point to a practical design: convolutions only in early blocks, much like the hybrid QuartzNet that kept learned convolutions in only two blocks.
A null result on all three checks is itself useful. It tells practitioners that, under self-supervision, unlabeled hours can replace the local module, and that the Conformer's extra cost buys nothing at scale.
References
- Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
- OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer
- Conformer without Convolutions
- Conformer: Convolution-augmented Transformer for Speech Recognition
- NEST: Self-supervised Fast Conformer as All-purpose Seasoning to Speech Processing Tasks
- Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
- Polynomial Mixing for Efficient Self-supervised Speech Encoders
- Scaling Effect of Self-Supervised Speech Models
- SUPERB: Speech Processing Universal PERformance Benchmark
- Robust Speech Recognition via Large-Scale Weak Supervision
- More Speaking or More Speakers?