Scaling Speech Recognition with Limited Transcribed Data

37 concepts36 papers1 proposal

The primary obstacle to deploying state-of-the-art Automatic Speech Recognition (ASR) is not a lack of audio, but a lack of transcribed audio. While raw speech is abundant, the manual process of mapping audio to text is notoriously expensive and time-consuming. This "transcription bottleneck" creates a significant barrier for specialized domains like particle physics Santamaría-Jordà et al., spontaneous lecture speech Akita et al., and endangered or low-resource languages Meelen et al..

To bypass this bottleneck, researchers have developed strategies to extract more value from untranscribed audio, leverage cheaper but noisier human labor, and even synthesize training data from text alone.

Learning from What the Model Already Knows

If a model is already partially trained, it can act as its own teacher. This is the core intuition behind semi-supervised training (SST). By running an existing ASR system over untranscribed audio, we generate "pseudo-labels."

However, pseudo-labels are inherently noisy. Visweswariah et al. noted early on that while unsupervised adaptation provides gains, a gap remains compared to supervised training. To bridge this, Veselý et al. proposed a "word selection" strategy: instead of training on the entire noisy transcript, the system only learns from words the model identifies with high confidence. This selective learning prevents the model from reinforcing its own errors, a technique that has proven particularly effective for low-resource languages like Somali Biswas et al..

Strategic Annotation via Active Learning

Since human transcription is the most expensive resource, we should only spend it on the most informative data. Active Learning (AL) shifts the focus from quantity to utility.

Drugman et al. demonstrated that by using confidence models to select only the most "difficult" utterances for human review, transcription costs can be significantly reduced without sacrificing performance. More recently, Hiruta et al. refined this by combining transcription uncertainty with "acoustic diversity." The intuition is that a model should not just ask for help when it is confused, but should also seek out audio samples that look different from anything it has seen before, ensuring a more representative training set.

Leveraging Low-Cost and Imperfect Sources

When professional transcription is too costly, researchers look for "good enough" alternatives. Novotney et al. showed that crowdsourcing via platforms like Amazon Mechanical Turk can produce ASR training data comparable to expert transcriptions at a fraction of the price. This approach has been successfully extended to under-resourced African languages, where Gelas et al. found that non-expert transcriptions were sufficient for ASR development.

Another strategy involves "harvesting" existing text that is loosely related to audio. Lecouteux et al. introduced "text island spotting," a method to find segments of speech that match imperfect text sources like closed captions or journalist prompts. Similarly, Akita et al. transformed formal written proceedings into spoken-style text to better match the spontaneous nature of lecture speech, effectively creating a language model without new transcriptions.

Synthesizing the Audio Gap

The most radical departure from manual transcription is removing the need for audio-text pairs entirely. If we have plenty of text in a new domain but no audio, can we simulate the missing link?

Bataev et al. proposed an end-to-end system that uses a text-to-mel-spectrogram generator to synthesize realistic audio features from text-only data. By using a GAN-based enhancer to make these features indistinguishable from real speech, the authors allow ASR models to adapt to new jargon or accents without ever hearing a human speak those words.

Improving the Human-in-the-Loop

Finally, rather than replacing humans, some research focuses on making human transcription more efficient. Jian Gao et al. introduced an "ML-in-the-loop" system that uses machine learning to flag likely human errors during the transcription process. By reducing human error by over 50%, they demonstrated that higher-quality data leads to a direct boost in ASR accuracy, maximizing the return on investment for every hour of manual labor. Similarly, Qamar et al. used Large Language Models to automatically review and correct information extracted from conversations, addressing the bottleneck at the post-processing stage.

Key Foundational Work

Exploiting large quantities of spontaneous speech for unsupervised training of acoustic models (2005) — contributed 1 concept elements to this direction.
Using quick transcriptions to improve conversational speech models (2004) — contributed 1 concept elements to this direction.
Automatic transcription of continuous speech using unsupervised and incremental training (2004) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_high_cost_of_transcribed_data_158f3439,
  title = {high cost of transcribed data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-cost-of-transcribed-data/rdr_158f3439ef4a4d208e46ae7ebcfb0081},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_high_cost_of_transcribed_data_158f3439_v1,
  title = {high cost of transcribed data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-cost-of-transcribed-data/rdr_158f3439ef4a4d208e46ae7ebcfb0081/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake