Teams building speech recognition for a regional dialect almost always work under a fixed budget of transcribed hours. They still have to decide how to spend it: record a handful of speakers at length, or record many speakers briefly. Neuroimaging has argued over the same choice for years, as "deep phenotyping" (few subjects, very dense data) versus population consortia (many subjects, sparse data). It has also learned that small cohorts produce inflated, non-replicating effects. Dialect speech recognition has not taken up either lesson. Its corpora are specified by total hours per region, and its adaptation gains are usually reported on a single split drawn from a few speakers.
We propose to measure, under fixed total hours, how dialect word error rate depends on the number of speakers $K$ and the hours per speaker $M$ as separate quantities. We will fit a two-factor scaling relation that predicts the best allocation for a new dialect from a small pilot. In the same resampling experiments we will also measure how much small-sample dialect adaptation gains shrink when a method is re-evaluated on new speakers of the same dialect. The practical deliverable has two parts: a corpus-design rule a field team can apply before recording, and an honest estimate of how much reported low-resource improvements are worth.
Core Claim
Holding total hours fixed, dialect error has two sources that respond to $K$ and $M$ differently. The first is between-speaker acoustic variation (voice, pitch, speaking rate). It shrinks mainly with the number of distinct speakers. The second is exposure to rare dialect words and constructions that the pretrained decoder tends to "correct" into the standard language. It shrinks mainly with total text volume and topical breadth. Deep single-speaker interviews can supply that breadth, so this term does not strictly require many speakers.
This leads to a concrete prediction: the best $K$ at a fixed budget is interior rather than "as many speakers as possible." It should shift toward depth when the dialect's lexical distance from the standard language is large. We summarize this with
$$ \mathrm{WER}(K, M) \approx E_\infty + a\,K^{-\alpha} + b\,(KM)^{-\beta}, $$
where $E_\infty$ is the irreducible error, $a K^{-\alpha}$ is the speaker-variation term, and $b (KM)^{-\beta}$ is the total-data term. The speaker term mirrors a random-effects variance that falls with the number of groups. The power-law data term is the usual empirical form of data scaling.
If recruiting a speaker costs $c$ hours-equivalent, the budget constraint is $K(M + c) = B$. The fitted law then gives the optimal split in closed form. The first claim is that this two-factor law predicts held-out-speaker error better than a one-factor law in total hours. The second claim is that allocations chosen from it beat the "maximize speakers" heuristic on dialects where lexical divergence is high.
Why This Is Not Already Done
The closest papers vary the amount of data, but never the speaker/depth split at fixed hours.
Overcoming Data Scarcity in Multi-Dialectal Arabic ASR via Whisper Fine-Tuning fine-tunes Whisper on fractions of Modern Standard Arabic and on pooled versus single-dialect data from the Massive Arabic Speech Corpus. It explicitly does not vary the number of speakers while holding hours fixed.
Swiss German Speech Translation and the Curse of Multidialectality varies which dialects are pooled. It finds that 10 extra minutes from other regions helps a monodialectal model by up to 6.45 BLEU, while full pooling hurts Valais by 3.37 BLEU. It studies translation rather than transcription, and it offers no predictive account of either effect.
Convex Low-resource Accent-Robust Language Detection in Speech Recognition shows that a convex dialect-detection head works down to 100 samples. It does not ask whether those 100 samples came from few or many speakers.
All three papers motivate this work: they establish that sample composition matters. None of them isolates speaker count.
The neuroimaging work plays a different role: it supplies the lens, not the method. The deep-phenotyping design in Scaling laws for language encoding models in fMRI used 3 subjects with 20 hours each and found logarithmic returns to per-subject data. The consortium design of Subcortical brain volume abnormalities in 2028 individuals with schizophrenia and 2540 healthy controls via the ENIGMA consortium pooled 4,568 people across 15 sites. Between them they frame the depth-versus-breadth choice we transplant. The same literature's attention to power and effect-size inflation motivates our replicability audit. We borrow no statistical machinery from these papers beyond speaker-level resampling.
Research Program
We run a grid of controlled fine-tuning experiments on corpora that record speaker identity and have enough speakers and depth to cover both ends of the design space.
ILSE (Automatic Speech Recognition for ILSE-Interviews) has 442 transcribed hours from 74 German participants, roughly six hours per person. It is the only corpus in our set where very deep allocations are possible, and it contains dialectal conversational speech.
GCND (Leveraging Geographic Metadata for Dialect-Aware Speech Recognition) has 411 hours of Flemish and southern Dutch dialect interviews. This supports many-speaker arms within a region. The file does not give speaker counts, and we will verify them before fixing the grid.
The Corpus of Japanese Dialects (Dialect Speech Recognition Modeling using Corpus of Japanese Dialects and Self-Supervised Learning-based Model XLSR) has over 60 hours of elderly speakers across 47 prefectures. It tests the same question on a spontaneous-speech corpus in a non-European language.
The Thai Dialect Corpus (Thai Dialect Corpus and Transfer-based Curriculum Learning Investigation) has 40 hours per dialect of read speech on prompts parallel to Central Thai. This gives a useful control. Because the text is shared across speakers, extra depth adds little new vocabulary, and the model predicts that the speaker term should dominate there.
For each dialect and each budget $B$ of 1, 4, and 16 hours, we sweep $K$ from 2 up to the maximum the corpus allows, with $M = B/K$. We draw training speakers at random and repeat each cell with at least five independent speaker draws. Every evaluation uses speakers disjoint from training.
We fine-tune two backbones, Whisper-small and XLS-R 300M, with a fixed recipe and a fixed step budget, so that only data composition varies. A secondary sweep with Whisper large-v3 on the 4-hour budget checks whether larger models shift the optimum toward more speakers. That would be the speech analogue of the dimensionality bottleneck hypothesized in the fMRI scaling paper.
We fit the scaling relation on the 1- and 4-hour cells and use it to predict the 16-hour cells. We also report two quantities per dialect:
- a speaker-term share, the fraction of reducible error attributable to $aK^{-\alpha}$;
- a lexical-divergence score, measured as the out-of-vocabulary rate of dialect transcripts under a standard-language lexicon.
We then test whether divergence predicts the optimal $K$ across dialects. The replicability audit reuses the same runs. For each small $K$, we pick the best of several adaptation configurations (learning rate, number of layers unfrozen, with and without dialect-identification multitask) on a dev set of two or three speakers. We then re-score the chosen configuration on fresh speakers. The gap between the selected dev gain and the fresh-speaker gain directly estimates winner's-curse inflation.
Baselines
Zero-shot reference. The no-adaptation point is zero-shot Whisper large-v3. On the Betthupferl Bavarian test set, A Multi-Dialectal Dataset for German Dialect ASR and Dialect-to-Standard Speech Translation reports it rising from 9% word error rate on Standard German to 31% on dialect speech, with about 63% error on dialect-specific words. We run it on every held-out speaker set, and we use Betthupferl only as an out-of-corpus test of German models trained on ILSE. Any allocation must beat this reference by a margin whose speaker-bootstrap confidence interval excludes zero.
Strongest existing adaptation recipe. The strongest existing adaptation method in the file is the multitask recognition-plus-dialect-identification XLS-R model of the Japanese dialect paper. It reports 38.1% character error rate on that corpus, and 37.0% when first adapted from the standard-Japanese Corpus of Spontaneous Japanese. We use it as the adaptation recipe inside every cell. It is the strongest competitor because it is the method a team would actually deploy, and our claim concerns how to spend data, not how to replace it. Fairness is structural: every arm gets the same recipe, steps, and tuning grid.
Allocation heuristics. The competing allocation rules are "maximize speakers," which sets $K$ at the largest value allowed with a minimum clip length, and "equal split," which sets $K = \sqrt{B}$ in 10-minute units. The fitted law beats them if, on held-out 16-hour cells and held-out dialects, its chosen allocation yields lower fresh-speaker error in a majority of dialects, with paired speaker-bootstrap intervals excluding zero.
Key ablation. The one-factor law $E_\infty + b(KM)^{-\beta}$ drops the speaker term. If it predicts held-out cells as well as the two-factor law, the core claim fails.
Pooling arm. We also run the pooled-dialect fine-tuning that the Arabic Whisper paper recommends. This checks whether borrowing neighbouring dialects substitutes for local speakers.
Why not just…?
Why not just record as many speakers as possible? This is the default intuition, and it is a genuine threat. If error falls monotonically in $K$ at every fixed budget, the law collapses to that rule. The evidence that it may not is indirect. Whisper's 63% error on dialect-only words shows that the dominant failure is lexical rather than acoustic, and brief recordings from many speakers may never expose rare words often enough. The $K$ sweep settles the question directly. We will report a monotone result as a finding, and the Thai parallel-prompt control is where we expect it.
Why not just report speaker-level confidence intervals and skip the allocation law? Reporting intervals is cheap and correct, and we adopt it everywhere. But intervals only say how uncertain a given corpus's result is. They do not tell a team how to collect the next corpus. The audit and the law answer different questions, and we share the same runs between them so the audit costs no extra compute.
Why not just select utterances from a larger pool, as in meta-reweighting? Domain-Aware Data Selection for Speech Classification via Meta-Reweighting reweights existing out-of-domain utterances for intent classification. It needs a large source pool and known domain labels, and it was not built for transcription. Our setting is prior to data existence, choosing what to record, so selection is complementary rather than competing. As a sanity check, we include a cheap acoustic-diversity selection arm over GCND utterances at fixed hours. This shows whether the speaker effect is really about speaker identity or merely about acoustic diversity.
Decisive Checks
The proposal succeeds if three things hold. First, the two-factor law beats the one-factor law on held-out 16-hour cells. Second, lexical divergence correlates with the optimal $K$ across the Dutch, German, Japanese, and Thai dialects, with Thai at the many-speaker end. Third, law-chosen allocations beat both heuristics on fresh speakers.
Independently of these, the audit is informative whatever it finds. It will report how much of a best-of-configurations dev gain on two or three speakers survives on new speakers. If shrinkage is large, gains reported from single small splits, such as the 10-minute improvement in Swiss German, should be read with correspondingly wide error bars.
References
- Overcoming Data Scarcity in Multi-Dialectal Arabic ASR via Whisper Fine-Tuning
- Swiss German Speech Translation and the Curse of Multidialectality
- Convex Low-resource Accent-Robust Language Detection in Speech Recognition
- Scaling laws for language encoding models in fMRI
- Subcortical brain volume abnormalities in 2028 individuals with schizophrenia and 2540 healthy controls via the ENIGMA consortium
- Automatic Speech Recognition for ILSE-Interviews: Longitudinal Conversational Speech Recordings Covering Aging and Cognitive Decline
- Leveraging Geographic Metadata for Dialect-Aware Speech Recognition
- Dialect Speech Recognition Modeling using Corpus of Japanese Dialects and Self-Supervised Learning-based Model XLSR
- Thai Dialect Corpus and Transfer-based Curriculum Learning Investigation for Dialect Automatic Speech Recognition
- A Multi-Dialectal Dataset for German Dialect ASR and Dialect-to-Standard Speech Translation
- Domain-Aware Data Selection for Speech Classification via Meta-Reweighting