Speech recognition for small languages and dialects now works mostly because of large models pretrained on huge piles of audio scraped from the web. A model learns general sound patterns from that audio, and then a few hours of transcribed target-language speech are enough to adapt it. The catch is that almost nobody can say what the pretraining audio was licensed for. As a result, the community tool built on top of such a model inherits a provenance question it cannot answer. That tool might be a fine-tuned model, a training recipe, or a packaged environment for field linguists. Reproducibility efforts in speech dodged the problem by shipping recipes and leaving users to supply their own licensed data and software. That is exactly the step that locks out under-resourced users.
This proposal measures the price of not dodging. We will pretrain the same self-supervised speech encoder on audio stratified by license provenance: public domain, then public domain plus openly licensed crowd recordings, then unaudited web audio of matched size. We then adapt each encoder to four real low-resource targets, with and without self-training on in-language unlabeled audio. The output is a "license price" curve: how many word-error-rate points a fully redistributable pipeline gives up relative to the unaudited one. We also test whether self-training on target-language audio, which communities can plausibly own, buys those points back. If the price is small, a field-linguistics tool can ship model, weights, code and pretraining data as one artifact with no user-supplied inputs. If it is large, we will know precisely where unaudited data is doing the work.
Core Claim
Our hypothesis has two parts. First, most of the benefit that large pretrained models give low-resource speech recognition comes from generic acoustic-phonetic structure, which clean audio can supply. Only a minority comes from the breadth and scale that currently requires unaudited web audio. Second, much of the remaining gap can be closed by pseudo-labeling, meaning a first model transcribes unlabeled in-language audio and is retrained on its own outputs. Self-training and Pre-training are Complementary for Speech Recognition showed this complementarity in English. There, pretraining on 53,000 hours of LibriVox, a public-domain audiobook collection, plus self-training reached 3.0% / 5.2% word error rate (WER) on LibriSpeech test-clean/test-other from just 10 minutes of labels.
That result already hints that a public-domain pretraining corpus can be strong. But it was never framed as a provenance claim, it was English-only, and it was never tested on noisy dialectal field audio. Testing it there is what is new.
How the Two Directions Meet
The licensing literature motivates the problem by showing that the standard workarounds hand the barrier back to the user. The Speech Recognition Virtual Kitchen (An Initial Prototype, The speech recognition virtual kitchen, turns one) packaged Kaldi and HTK recipes in virtual machines and later in Vagrant configuration files. Its install wrappers asked users to fetch restricted toolkits and corpora themselves. The OSIRRC 2019 jig did the same for information retrieval with Docker, mounting user-supplied licensed collections at run time. Plummer and Beckman showed the end state for low-resource users: a synthesis VM depending on MATLAB could not legally be redistributed, and large images broke on student laptops. Olive found the same legal block for long-term preservation.
All of these treat licensing as a packaging problem. We treat it as a data-selection problem upstream of the model. If every input is redistributable, the decoupling step disappears.
Blameless Users in a Clean Room proves that formal copy protection via differential privacy holds only for a "golden" training set purged of unauthorized copies. It also argues that such sets are near-impossible at foundation scale. That paper constrains our framing. We do not claim formal clean-room guarantees, which were defined for generative text and image models. Instead we use provenance licensing as an operational proxy for a golden set. We then ask the empirical question the paper leaves open: how much downstream accuracy a speech model loses when restricted to such a set.
The data-scarcity literature supplies the method and the evaluation regime. Omnilingual ASR is the unaudited reference point. It scales a self-supervised encoder of the wav2vec 2.0 type up to 7B parameters on 3.84M hours, which is mostly aggregated web audio. Our controlled tiers are designed to isolate what that aggregation buys.
Jones et al. supply the starkest target. On noisy Čakavian dialect field interviews, a fully open Kaldi pipeline wrapped in Elpis reached a median WER of 129%. A fine-tuned Whisper large-v3 reached 35.48%.
Research Program
We will build four pretraining corpora and fix the hours across tiers so that provenance is not confounded with quantity.
The first tier is public domain only, drawn from LibriVox and its multilingual derivative MLS. The second tier adds Common Voice crowd recordings under their open license, which brings in many more languages. The third tier is a matched-hours sample of unaudited crawled web speech of the kind aggregated in AllASR. The fourth is a provenance-flagged "derived" condition, explained below.
A fourth axis matters because language diversity and license cleanliness are tangled together. Public domain is English-heavy, whereas Common Voice is broad. We therefore also run tier two at matched language count against tier three.
On each corpus we pretrain a wav2vec 2.0 encoder with about 300M parameters. That size is chosen deliberately: the fine-tuned model must run on the commodity hardware that Elpis and Plummer and Beckman target. All tiers use the same steps, batch size and learning-rate schedule.
Each encoder is then adapted to each target in two ways. The first is plain supervised fine-tuning with a character-level alignment-free (CTC) loss. The second follows the recipe of Xu et al.: fine-tune, decode the target's unlabeled audio with a small in-language language model, retrain on the pseudo-labels, and repeat once.
The derived condition tests whether self-training can launder an unaudited model. Here the pseudo-labels come from fine-tuned Whisper rather than from our clean model. We report it but treat its provenance as unresolved, not clean. A teacher's outputs plausibly carry its training data's legal status, and we want to measure the accuracy temptation honestly.
Finally, we package the best fully clean configuration as a single container. It ships code, weights, pretraining manifest and a redistributable demo corpus. We verify that it trains and decodes on a laptop-class machine with no user-supplied credentials or data. This is the concrete artifact that the decoupled-jig design cannot produce.
One honesty point: the target-language corpora also have provenance. MinSpeech, for example, is built from TV dramas. Our provenance tiers apply to pretraining. Target data enters as the benchmark supplies it, and the shipped demo artifact uses only target audio whose license permits redistribution.
Baselines
The naive reference point is training from scratch with an open classical toolkit. On Čakavian this is reproduced directly by the Elpis Kaldi pipeline of Jones et al., which reported a median WER of 129% and an average of 130.1%. On MinSpeech it is the Kaldi chain model of Lin et al., which reported 27.86% character error rate (CER). Any pretrained tier should crush this. If it does not, our training is broken.
The strongest competitor on the main question is fine-tuning a large unaudited model on the same target data. For Čakavian this is Whisper large-v3 fine-tuned on the identical 5.7 hours: median WER 35.48% and average 57.3%, as reported by Jones et al.. We will rerun it ourselves with the same splits and segmentation, because they found Whisper very sensitive to chunking. We will also fine-tune the released Omnilingual 300M checkpoint with the same hyperparameter search budget as our encoders. The reason is that a size-matched comparison is fairer than one against a 7B model. No reported result was found on Lacuna for Omnilingual ASR on Čakavian, MinSpeech, KSC2 or Kichwa, so these numbers will come from our runs.
For Kichwa the strongest reported system is the mms-300 acoustic model with Llama-3-8B in-context rescoring, at 17.00% WER versus 17.72% for an n-gram model, from In-context Language Learning for Endangered Languages. For MinSpeech it is HuBERT-large with hybrid CTC/attention at 20.21% CER from Lin et al., although that number uses the full 1,778 labeled hours, whereas we subsample. For KSC2 it is the ESPnet Transformer at 15.6% overall WER, which degrades to over 27% on spontaneous speech, from Mussakhojayeva et al..
Fairness rules are fixed in advance. Every system sees the same labeled hours, the same unlabeled audio, the same language model text and the same hyperparameter trials. Results count as "beating" the unaudited baseline if the clean tier with self-training is within two absolute WER points of the size-matched unaudited tier on at least three of the four targets. They count as "beating" Whisper if the clean tier comes within five points of its median WER on Čakavian. The key ablation is the clean tier without self-training, which shows whether self-training, rather than pretraining choice, closes the gap.
Why not just…?
Why not just fine-tune the open-weight Omnilingual or Whisper checkpoints and rely on their licenses? This is the real threat, and for many users it may be the right answer. A permissive license on weights says nothing about whether the training audio was cleared, though. Cohen et al. argue that models trained on uncurated data are exposed in ways their licenses do not resolve (Blameless Users). Our experiment does not argue around this alternative; it prices it. If the matched-tier gap turns out large, the honest conclusion is that communities currently need unaudited weights, and we will say so with numbers.
Why not just use fully open classical toolkits like Elpis and Kaldi? They are clean and laptop-friendly but unusable on field audio. The 129% median WER on Čakavian (Jones et al.) is far above the roughly 30% threshold those authors cite for useful documentation support. We include this pipeline only as the floor.
Why not just synthesize training data with multi-speaker text-to-speech? The synthesizer itself is trained on high-resource voice corpora whose licenses then flow into the synthetic set, so this moves the provenance question rather than removing it. Synthetic-only training also still lags real speech: Casanova et al. report 33.96% WER for Portuguese and 36.59% for Russian. We do not pursue this route.
Why not just pretrain on English LibriVox, as Xu et al. already did? This is our tier one, so the proposal tests it directly. The open question is whether an English-dominant public-domain encoder transfers to Southern Min tones or Kazakh–Russian code-switching. The matched-language-count comparison of tiers one and two answers that.
Proposed Experiments
The four targets span the regimes in the file. The first is Čakavian, with 5.7 hours of training audio and noisy, overlapping interview test audio. The second is MinSpeech Southern Min, subsampled to 10 and 100 labeled hours, with its 2,237 unlabeled hours available for self-training. The third is KSC2 Kazakh, subsampled to 10 labeled hours, with the remaining audio treated as unlabeled and results reported separately on the read and spontaneous subsets. The fourth is Kichwa, from the endangered-language rescoring benchmark, where about 2 hours of training audio are available. We report WER, or CER for Southern Min, on each target. We also report the license price, defined as the clean tier's error minus the matched unaudited tier's error, both with and without self-training. Finally, we report the fraction of the clean-to-unaudited gap that self-training recovers.
Three checks decide the claim. The first: if self-training shrinks the license price on noisy Čakavian and spontaneous Kazakh as well as on read speech, the hypothesis survives on the hardest audio. If the price shrinks only on read speech, the clean recipe is limited to easy conditions, and we will report that. The second concerns the derived teacher condition. If pseudo-labels from Whisper recover substantially more than clean self-training, we learn that the clean deficit is about label quality rather than representations, which redirects future work toward better clean teachers. The third: the packaged clean artifact must train and decode on a laptop-class machine end to end with no external inputs. This is the property that every decoupled design in the licensing literature lacks.
References
- Self-Training and Pre-Training are Complementary for Speech Recognition
- The Speech Recognition Virtual Kitchen: An Initial Prototype
- The speech recognition virtual kitchen
- The speech recognition virtual kitchen turns one
- Overview of the 2019 Open-Source IR Replicability Challenge (OSIRRC 2019)
- Sharing Speech Synthesis Software for Research and Education Within Low-Tech and Low-Resource Communities
- Collaborating with executable content across space and time
- Blameless Users in a Clean Room: Defining Copyright Protection for Generative Models
- Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages
- Comparing Kaldi-Based Pipeline Elpis and Whisper for Čakavian Transcription
- Elpis, an Accessible Speech-to-Text Tool
- MinSpeech: A Corpus of Southern Min Dialect for Automatic Speech Recognition
- In-context Language Learning for Endangered Languages in Speech Recognition
- KSC2: An Industrial-Scale Open-Source Kazakh Speech Corpus
- TTS applied to the generation of datasets for automatic speech recognition