Using the Untranscribed Backlog's Own Noise to Make Fieldwork ASR Robust

Language documentation projects typically hold hundreds of hours of recordings and only a few hours of transcripts. Transcribing one hour of spontaneous fieldwork speech takes a trained linguist 30 to 50 hours. Speech recognizers built on self-supervised models, meaning networks pre-trained on large amounts of unlabeled audio, can now be fine-tuned on under an hour of transcribed speech. They produce drafts that a linguist corrects instead of typing from scratch. These drafts break down in exactly the conditions that matter most: a new speaker, a different recording session, or a village recording with animals, chatter, or wind. This proposal starts from a simple observation. The untranscribed backlog that creates the bottleneck also contains the specific noise the recognizer will face, recorded by the same microphone in the same place. We propose mining the silent gaps in that backlog to build a noise bank for the target site. We then self-train the recognizer on its own transcriptions of the backlog, with a consistency objective. Under that objective, adding site-matched noise to an utterance must not change the model's internal representation or its output.

In two sentences: current fieldwork ASR uses the untranscribed hours for nothing, or at most for pseudo-labels, and treats noise with generic augmentation corpora or not at all. We claim the backlog should be used twice, as unlabeled speech and as a site-matched noise source. This combination should shrink the gap in accuracy and correction effort on held-out speakers and noisy sessions more than either generic noise augmentation or plain self-training does.

Core Claim

The claim is empirical and falsifiable. Take a multilingual self-supervised recognizer adapted to 10 to 60 minutes of transcribed fieldwork speech. Noise-consistent self-training on the project's own untranscribed audio, using noise harvested from that audio's non-speech regions, will reduce character error rate and post-editing effort on held-out speakers and noisy sessions. The comparison is against three alternatives given identical data and compute: fine-tuning alone, fine-tuning with generic noise augmentation, and self-training without noise. The ingredient we isolate is where the noise comes from. If generic noise at matched signal-to-noise ratio (SNR) gives the same gains, the claim is false, and the contribution shrinks to a careful robustness evaluation of fieldwork ASR.

Where the Gap Sits

Four strands of prior work motivate the proposal, and each leaves the same hole.

Breaking the Transcription Bottleneck fine-tunes two models, MMS-1B (with small added adapter modules and a frozen base) and XLS-R-300m, on 10 to 120 minutes from five endangered languages. It finds gains plateau after roughly 60 minutes and error rates of 30–38% on tone, nasality, and length. It used overlapping speakers between train and test and never tested noise, so its numbers are likely optimistic for real deployment.

Enenlhet as a case-study supplies the evidence for that suspicion. With leave-one-speaker-out evaluation, Whisper and XLS-R-53 both show a large generalization gap, and error rises with acoustic sharpness. About 110 of that corpus's 120 hours remain untranscribed and unused.

Speech Recognition Corpus of the Khinalug Language reports the same collapse on unseen speakers with 2.67 hours of data. It also shows that adding a related language (Azerbaijani) hurts, which constrains us: we do not rely on cross-lingual co-training. The Yoloxóchitl Mixtec study notes that its Conformer degrades sharply on recordings with animal sounds and chatter, where human transcribers stay robust.

Two papers come closest to our method, and they show why the obvious combination is not enough. A Unified Denoising and Adaptation Framework for Bengali Dialectal ASR fine-tunes WavLM, a model pre-trained with a denoising objective, on 94 hours of standard Bengali. It then adapts to dialects while mixing in NOISEX-92 babble, factory, and vehicle noise. That is "self-supervised model plus noise augmentation for a low-resource variety," but it depends on a high-resource national standard and synthetic noise, and never uses untranscribed target audio.

MSDA combines contrastive self-supervision across domains with meta pseudo-labeling for unsupervised domain adaptation, but only between Greek broadcast, parliament, and podcast domains. It has abundant labeled source data and does not target noise explicitly.

Three further papers supply parts of our method. Mixup-breakdown trains a student separation model to match a teacher on remixed out-of-domain mixtures. We borrow the teacher–student remix consistency and apply it to recognition, with noise sources taken from recording gaps rather than from a separation model. Bring the Noise (Cleancoder) denoises the internal layers of a pretrained Conformer but needs paired clean and noisy audio, which fieldwork never has. Our consistency target is "the recording as captured" versus "the recording with extra site noise," so no clean reference is needed. Self-training and Pre-training are Complementary establishes that pseudo-labeling helps on top of self-supervised pre-training, but in high-resource English. The known risk is that the student learns the teacher's mistakes ("confirmation bias"), which our filtering has to manage with only minutes of labeled data.

Research Program

The system has three stages, all run per language project.

Stage 1: build the noise bank. First, harvest noise from the untranscribed backlog. A voice activity detector finds non-speech stretches longer than one second. We then embed each stretch with a lower layer of the self-supervised model, since lower layers keep acoustic and channel detail, as Exploring wav2vec 2.0 on Speaker Verification and Language Identification observed. Next, we cluster the embeddings into environment types. Finally, we discard stretches the recognizer transcribes with confident characters, which removes hidden speech and near-babble. Removing near-babble matters because speech-like noise is where quality estimators fail, per Specialized Speech Enhancement Model Selection. We also estimate each session's SNR so that augmentation SNRs can be drawn from the corpus's own distribution rather than a fixed range.

Stage 2: train the teacher. The teacher is MMS-1B with adapters, fine-tuned on the transcribed minutes as in Liang et al. It transcribes the backlog, and we keep utterances whose average CTC confidence (the per-frame probability of the chosen character) exceeds a threshold set on the small labeled dev set.

Stage 3: train the student. The student is initialized from the teacher and trained on labeled data plus pseudo-labeled utterances. Each pseudo-labeled input $x$ is remixed as $x+n$, where $n$ is drawn from the bank, excluding the utterance's own session. The student minimizes

$$\mathcal{L} = \mathcal{L}_{\text{CTC}}(x_\ell, y_\ell) + \lambda\, \mathcal{L}_{\text{CTC}}(x+n, \hat{y}) + \mu \sum_{k \in K} \lVert h^s_k(x+n) - \mathrm{sg}[h^t_k(x)] \rVert_1 .$$

Here $(x_\ell, y_\ell)$ are transcribed pairs, $\hat{y}$ is the teacher's pseudo-label, and $h^s_k$ and $h^t_k$ are student and teacher hidden states at layer $k$ from a chosen set $K$ of middle layers. $\mathrm{sg}$ stops gradients to the teacher, and $\lambda, \mu$ are weights. We run two teacher–student rounds.

The consistency term is the part that goes beyond ordinary pseudo-label augmentation. It pushes the representation, and not only the output, to ignore site noise. It is also where Cleancoder's paired clean reference is replaced by the recording itself.

Evaluation corpora. We use the Enenlhet corpus (about 5 hours transcribed, about 110 hours untranscribed, speaker metadata available), the Khinalug corpus, and the five Liang et al. languages, all subject to access. We also use Yoloxóchitl Mixtec, whose 100 expert-transcribed hours let us simulate scarcity. We keep 10 to 60 minutes as labeled, treat the rest as the backlog, and still have ground truth to score it. Its novice transcriptions support a correction-workflow evaluation in the spirit of the Novice Transcription Correction task.

Baselines

Do nothing. The reference point is zero-shot transcription with Whisper (Robust Speech Recognition via Large-Scale Weak Supervision) and with MMS. Whisper's scale gives robustness to common noise, but it is known to degrade and hallucinate on languages with no web presence. This baseline shows how much of the bottleneck any adaptation removes. For context, Tusom2021 reports 71.9% phone error rate zero-shot for a language-independent phone recognizer, and 33% after full fine-tuning, on East Tusom.

Strongest existing method. The main competitor is the Liang et al. recipe: MMS-1B adapter fine-tuning and XLS-R-300m full fine-tuning on the same 10, 30, and 60 labeled minutes. It is the most directly comparable published approach for this data regime. The only numbers in the file are its 30–38% error rates on phonological features, measured with speaker overlap. No reported word or character error rate under held-out-speaker or noisy-session conditions was found on Lacuna for these languages. We will therefore rerun the recipe ourselves under our splits. We will also run Whisper-medium fine-tuning, which outperformed XLS-R-53 on Enenlhet.

Generic noise augmentation. This is the Bengali-paper approach without its unavailable 94-hour stage: WavLM-Large and MMS fine-tuned on the labeled minutes with NOISEX-92 and similar generic noise, at SNRs matched to our corpus-estimated distribution.

Plain self-training. This is standard pseudo-labeling on the same filtered backlog, with no noise remixing and no consistency term.

Continued pre-training. Here the self-supervised objective continues on the backlog before fine-tuning.

Consistency-based adaptation. This is an MSDA-style adaptation that uses the backlog as the target domain.

Ablations. Four ablations isolate the key ingredient. First, harvested noise versus generic noise, holding everything else fixed, including SNR distribution, number of mixtures, and training steps. Second, the consistency term switched off. Third, noise taken from the same session versus other sessions, to check for leakage. Fourth, a shuffled noise bank taken from a different language project, which tests whether "matched" noise matters or just any real field noise.

Fairness. All systems get the same labeled minutes and the same backlog where applicable. They also get the same GPU-hour budget, and the same number of hyperparameter trials tuned on the same small labeled dev set, which has no test speakers. Beating the strongest baseline means a statistically significant character error rate reduction over Liang-recipe MMS on the held-out-speaker and noisy-session splits, bootstrap over utterances with p < 0.05, in at least four of the six languages. It also requires that the gain over generic augmentation and over plain self-training survives separately.

Why not just…?

Why not just add generic noise augmentation during fine-tuning? This is the most serious threat, and the Bengali paper shows it helps for synthetic noise. Fieldwork noise differs in two ways: its spectrum depends on the site (animals, rain on roofs, specific recorders), and it is often non-stationary. Generic banks may miss both. We do not argue the point in advance. The harvested-versus-generic ablation at matched SNR decides it. If the two tie, we will report that generic augmentation suffices and redirect the contribution to the evaluation protocol.

Why not just put a speech enhancement front end before the recognizer? Enhancement models overfit to their training noise and fail on novel noise spectra. They also tend to remove the cues this data depends on, such as nasality, glottal stops, and length contrasts, which are already the recognizer's weakest points. Cleancoder-style internal denoising needs paired clean audio. We include an off-the-shelf enhancer as an extra baseline and report phonological-feature error separately, so any damage to those cues becomes visible.

Why not just do ordinary self-training, or continued pre-training on the backlog? Both use the untranscribed hours, and either could capture most of the gain through speaker diversity alone. The complementarity result supports trying them, and both are baselines. Our decomposition separates the sources: plain self-training measures the speaker and diversity gain, and the noise and consistency terms measure what is added on top. If the added terms contribute nothing on noisy sessions, the claim fails.

Why not just use Whisper, since it is already noise-robust? Its robustness comes from web-scale data these languages lack. The Enenlhet study shows even fine-tuned Whisper leaves a large held-out-speaker gap. We keep it both as the zero-shot reference and as a fine-tuned baseline.

Proposed Experiments and Decisive Checks

We use three splits per language. The first is held-out speakers (leave-one-speaker-out, as in the Enenlhet study). The second is held-out sessions. The third is a strict unseen-environment split: we hold out a whole noise cluster and remove its sessions from both the noise bank and the pseudo-label pool. This last split separates realistic transductive use from true generalization to unheard noise. Using the backlog's own sessions is legitimate in deployment, because the backlog is the target, but it must be reported separately.

The metrics are character and word error rate, stratified by estimated session SNR, together with phonological-feature error following Liang et al. We also measure post-editing effort in two ways. The first is character edit distance between the draft and the reference, used as a proxy. The second is a small timed correction study with native-speaker transcribers, following the protocol of Evaluating Workflows for Creating Orthographic Transcripts. On Mixtec, we additionally fuse drafts with novice transcriptions using ROVER, a word-level voting scheme across transcripts, to test whether robustness gains carry through to the correction workflow. As a sanity check that the method does not overfit to one site, we run the adapted English analogue on a subset of Speech Robust Bench corruption categories, using its normalized WER degradation. This matters because that benchmark found wav2vec 2.0 and MMS vulnerable to natural noise.

Three results would be decisive. The first is whether the harvested-minus-generic difference is positive on the noisy-session and unseen-environment splits. The second is whether the consistency term adds gains beyond noisy pseudo-labels alone. The third is whether the error-rate gains turn into measurably shorter correction time; without that, the transcription bottleneck has not moved.

A negative first result, together with a positive self-training result, would still be useful to documentation practitioners. It would say that the untranscribed backlog is worth using for pseudo-labels, but that site-matched noise is not worth the extra effort to harvest.