When the Transcriber Is Also Out of Date: Self-Updating Speech Recognition for Spoken News in Low-Resource Languages

Language models stop learning when training ends, so they confidently give stale answers about events after that date. The standard remedy is to retrieve fresh documents at question time. That remedy assumes fresh documents exist as written web text. For Hausa, Yoruba, Igbo and many other languages, recent news mostly travels by voice: community radio, voice notes, public announcements. The natural pipeline is to transcribe those broadcasts with automatic speech recognition (ASR, software that converts speech audio to text), index the transcripts, and let a language model retrieve from them.

This note argues that the naive pipeline has a specific flaw. The speech recognizer is out of date in the same way the language model is. The words that carry a new fact are often the words a recognizer trained before that fact cannot spell: the name of a newly appointed official, a new disease, a new drug, a new place in the news. These are exactly the words retrieval and answering depend on. We propose to measure this failure and to fix it with a loop that needs no new labeled audio. A new term that matters this week is said many times, by many speakers, across many broadcasts. That repetition is a signal for discovering the term's spelling, adding it to the recognizer's vocabulary, re-transcribing the archive, and re-indexing.

Core Claim

The claim has two parts, and both can be tested. The first is diagnostic. In a spoken-news retrieval pipeline for a low-resource language, overall word error rate (WER, the fraction of words transcribed wrongly) badly underestimates downstream failure. Errors concentrate on post-cutoff named entities and terms, and temporal question-answering failures concentrate on questions whose answer or key term is one of those entities. The second is constructive. A weekly loop can recover much of that loss without new human transcription. The loop mines candidate new terms from low-confidence transcript regions, keeps only those that recur consistently across independent recordings, and feeds them back into the recognizer as a biasing vocabulary (a list of words the decoder is nudged toward).

The work is meant to be distinct from three bodies of prior work. Retrieval against obsolescence, such as RealTime QA, retrieves from live English web text, and multilingual editing, such as ReMaKE, retrieves from clean curated fact triples. Both assume that the text of the update is already correct. Low-resource speech work such as NaijaVoices and AfriSpeech-200 measures transcription accuracy on fixed test sets with no time dimension. Spoken retrieval work such as WavRAG works in English over a fixed collection. None of them treats the recognizer's own vocabulary as stale knowledge that must be updated together with the index.

Why This Gap Is Real

Several papers motivate the proposal. Nouza et al. transcribed more than 100,000 hours of Czech radio for keyword search, which shows that broadcast-scale transcription is feasible. That work was archival, not used to keep a question-answering system current. RealTime QA shows how much fresh retrieval matters. GPT-3 with Google search reached 66.5% multiple-choice accuracy against 44.9% closed-book. The same paper attributes 34% of errors to retrieval failures, and that was on clean English text, so noisier sources should make retrieval worse. NaijaVoices shows how fragile off-the-shelf recognition is in these languages: Whisper Small scored 193.01% WER on Hausa before fine-tuning and 21.00% after.

Other papers constrain the design. Fleurs-SLU finds that cascades, meaning ASR followed by a text language model, are generally more robust than end-to-end speech models on lower-resource languages. That pushes us toward a cascade even though WavRAG argues cascades should be replaced. WavRAG's audio-to-text retriever was trained on roughly 1.5 million paired samples, which these languages do not have. AfriqueLLM finds that continued pretraining on monolingual low-resource text erodes general abilities. That argues against putting fresh facts into language-model weights.

Two papers supply methods. ReMaKE gives the retrieve-and-prepend recipe for updating facts across languages, and its reliability and locality measures. We adapt locality to ask whether added vocabulary corrupts transcription of unrelated speech. AURORA treats index maintenance under vocabulary drift as continual learning. It does so only after text exists; we move the drift problem one stage earlier, into the recognizer.

Research Program

The system has three parts.

Base cascade. A fine-tuned Whisper or MMS recognizer for Hausa, Yoruba and Igbo, trained on NaijaVoices, transcribes a weekly stream of broadcasts. A multilingual dense retriever in the style of ReMaKE indexes the transcripts in time-stamped chunks. An instruction-tuned language model answers questions using retrieved chunks and the current date.

Weekly term discovery. Each week we take spans where the recognizer's confidence is low or its alternative hypotheses disagree. We cluster them across recordings by acoustic similarity and by similarity of their alternative spellings. Clusters that appear in several independent broadcasts and speakers within a short window become candidate new terms. For each cluster we choose a spelling. One option is consensus over the recognizer's alternative outputs. The other is matching against any thin text signal available that week, such as target-language headlines or English news names transliterated by rule.

Feedback. Accepted terms enter a contextual-biasing list during decoding. The recent archive is re-decoded and the index rebuilt. A term is removed when its recurrence fades, so the list stays short.

The scientific content lies in two places. One is the diagnostic decomposition: how much of the question-answering loss comes from novel-entity transcription, as opposed to retrieval or reading. The other is a precision study of the recurrence filter. The loop has a hazard: adding wrong terms can make the recognizer hallucinate them into unrelated speech. We measure that directly as a vocabulary-locality score.

Baselines

The naive reference point is the closed-book language model with the current date in the prompt. RealTime QA reports that date prompts hurt closed-book models, and that closed-book GPT-3 reached 44.9% against 66.5% open-book on English. Low-resource closed-book numbers for our setting: no reported result found on Lacuna.

The strongest straightforward competitor is the static cascade. It uses the same NaijaVoices-fine-tuned recognizer, frozen at a cutoff date, with the same retriever and the same reader over the same transcribed broadcasts. Only the feedback loop is missing. The best recognizer number in the file is Whisper Small at 21.00% WER on Hausa after NaijaVoices fine-tuning (NaijaVoices). Spoken temporal question answering in these languages: no reported result found on Lacuna. We will therefore establish the static cascade's number ourselves. Beating it means a statistically significant gain in answer accuracy on the subset of questions involving post-cutoff entities, with no more than one point of loss on questions about older entities.

The second strong competitor is the static cascade with phonetic fuzzy matching. Queries and transcripts are both mapped to a phonetic or character-n-gram representation, so garbled spellings of a new name can still match. This tests whether retrieval that tolerates errors makes recognizer updating unnecessary.

The third competitor is retrieval over machine-translated English news, using the same reader. This tests whether local audio is needed at all.

The fourth is an end-to-end speech language model, such as the Gemini 2.0 Flash class evaluated in Fleurs-SLU, queried over chunked audio. We will report it with its length limits noted.

Two ablations isolate the key ingredient. The first adds all low-confidence candidates without the cross-recording recurrence filter; it shows whether repetition is what makes the loop safe. The second replaces discovery with an oracle list of the true new entities; it bounds what perfect discovery could achieve. An expensive reference arm fine-tunes the recognizer each week on a small budget of human-transcribed new audio (one hour per week).

All arms share the same audio, retriever, reader and question set. All tuning uses the same development weeks, separate from the test weeks. The reader's compute is fixed across arms, and only the recognizer stage differs.

Why not just…?

Why not just translate English news into the target language and retrieve over that? For national and international stories this may work well. RealTime QA itself notes that its sources lean heavily toward US and UK events. The local facts that matter here, such as a state health advisory or a local appointment, are often never covered in English. NaijaVoices also warns that translated text brings translationese. We do not assume the answer. Every question is labeled by whether its event appeared in English outlets that week, and the machine-translation arm is scored on both subsets. If it matches our system on locally sourced questions, the proposal's premise is wrong.

Why not just make retrieval tolerant of spelling errors instead of fixing the recognizer? This is the most serious threat. Fuzzy phonetic matching may recover much of the retrieval loss. The weakness we expect is at answer time. If the recognizer writes a new minister's name three different wrong ways, retrieval may still find the passage, but the reader then copies a wrong spelling or merges two people. We therefore report retrieval recall and answer exact match separately. If fuzzy matching closes the gap on both measures, we will recommend it and present the loop as unnecessary.

Why not just fine-tune the recognizer periodically on newly transcribed speech? It would work if labels were available, which is why it appears as a reference arm. Weekly human transcription in these languages is the bottleneck the whole area faces; Klehr et al. show that machine pre-transcription reduces human effort but does not remove it. The comparison shows how much of that arm's benefit the label-free loop recovers.

Why not just use an end-to-end speech model and skip transcripts? Fleurs-SLU finds cascades generally more robust for lower-resource languages. Its findings also point to input-length limits that rule out reading hours of broadcast at once. We include the arm anyway, because the benefit of transcription may shrink as these models improve.

Proposed Experiments

The first experiment is a controlled simulated cutoff on existing data. In NaijaVoices and in the clinical portion of AfriSpeech-200, we choose a set of named entities, and drug names for the clinical portion. We remove every utterance containing them from recognizer training, then treat their test utterances as "post-cutoff". Templated questions are built from the sentences containing those entities. This gives exact ground truth and isolates entity novelty from everything else. We report overall WER, novel-entity recall, retrieval Recall@10, and answer exact match for every arm.

The decisive diagnostic check is whether novel-entity recall collapses while overall WER barely moves, and whether the static cascade's question-answering failures concentrate on those entities. If they do not, the problem is mis-located and the loop is moot.

The second experiment runs on real data over time. Working with Nigerian community radio partners, we collect about twelve weeks of time-stamped broadcasts in Hausa and Yoruba. Native-speaker annotators write weekly questions about events reported that week, following the RealTime QA protocol, each labeled as local-only or also covered in English. The first four weeks are for development and the remaining eight for testing.

Three outcomes would settle the question:

  1. Gains on post-cutoff-entity questions over the static and fuzzy-matching cascades.
  2. Vocabulary locality: WER on held-out older speech stays within one point when biasing terms are added.
  3. The no-filter ablation degrading locality while the filtered loop does not.

If the oracle-list arm is far above the loop, the remaining work is discovery. If both sit near the static cascade, the bottleneck is retrieval or reading, and the claim fails.