Multi-Layered Pipelines for Speech Corpus Curation

16 concepts13 papers1 proposal

Building a text corpus is relatively straightforward: you scrape web pages, clean the formatting, and tokenize the characters. Building a speech corpus, however, is fundamentally different. Audio is continuous, noisy, and packed with non-verbal information like speaker identity, emotion, and environmental acoustics. Furthermore, human speech rarely matches formal written grammar; it is full of phonetic reductions, regional accents, and colloquial variations.

To bridge the gap between raw audio and machine learning models, modern speech processing relies on structured, multi-layered curation pipelines. These pipelines automate the ingestion, filtering, and alignment of audio while generating rich, multi-tiered annotations. By decoupling the acoustic signal into parallel layers of information, these systems allow models to learn exactly what was said and how it was said.

The Anatomy of an Automated Speech Pipeline

When scaling datasets to thousands of hours, manual curation becomes impossible. Instead, researchers deploy multi-stage automated pipelines to transform raw, "in-the-wild" audio into clean training data.

[Raw Audio Source] 
       │
       ▼
┌──────────────────────────────┐
│  Quality Filtering & De-noising│  <-- SNR, Bandwidth, VAD
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│   Segmentation & Alignment   │  <-- Forced Alignment, Diarization
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│  Multi-Layered Transcription │  <-- Phonetic, Orthographic, Metadata
└──────────────────────────────┘

The first challenge is audio quality. Raw internet audio or broadcast streams are often riddled with background noise and low-bandwidth recordings. To address this, the pipeline developed by Wenbin Wang et al. for GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speech filters large-scale datasets by calculating signal-to-noise ratios (SNR) and bandwidth, discarding low-quality clips before applying speech enhancement. Similarly, Jiayan Lin et al. filter audio based on signal energy and SNR in MinSpeech: A Corpus of Southern Min Dialect for Automatic Speech Recognition.

Once the audio is clean, it must be sliced into manageable chunks. This requires matching continuous audio to text transcripts, a process known as forced alignment. In the GigaSpeech pipeline, Guoguo Chen et al. utilize multi-stage forced alignment in GigaSpeech: An Evolving, Multi-Domain ASR Corpus with 10, 000 Hours of Transcribed Audio to segment long audio streams and validate them against common transcription errors. For web videos, Salvatore Carta et al. align audio chunks directly with subtitle timings in SardinianVoxes: A Speech Recognition Dataset for the Sardinian Languages.

Finally, to handle multi-speaker environments, pipelines like Emilia, introduced by Haorui He et al. in Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation, incorporate vocal separation and speaker diarization. This ensures that the resulting segments contain distinct, clean speech from individual speakers rather than overlapping voices.

Why Single-Text Transcripts Fail

If a speech dataset only contains a standard text transcript, downstream models struggle. For instance, if a speaker says "gonna" but the transcript reads "going to", an Automatic Speech Recognition (ASR) model receives conflicting signals about how those phonemes sound.

To solve this, modern pipelines employ a dual-layer transcription strategy. As demonstrated by Tahir Javed et al. in IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages, datasets can provide both a verbatim phonetic utterance (exactly what the speaker's mouth produced) and a standardized grammatical form. This dual-layer approach allows speech recognition models to learn natural, messy acoustic patterns while remaining fully compatible with downstream text-processing systems that expect formal grammar.

Audio Signal:  [ ♫ "wanna go" ]
                     │
                     ▼
Layer 1 (Phonetic):  "wanna go"       <-- Trains the Acoustic Model
Layer 2 (Grammar):   "want to go"     <-- Aligns with Downstream NLP

This multi-tier strategy expands significantly depending on the target application:

Balancing Scale and Nuance

The design of a curation pipeline depends heavily on whether the goal is massive scale or high-fidelity nuance.

For large-scale speech recognition and generation, pipelines prioritize automation. The authors of GigaSpeech: An Evolving, Multi-Domain ASR Corpus with 10, 000 Hours of Transcribed Audio and Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation demonstrate that fully automated pipelines can process tens or hundreds of thousands of hours of audio, capturing human nuances like breathing and emotional variations directly from "in-the-wild" data.

Conversely, for speech synthesis (TTS), emotional modeling, or documenting low-resource languages, pipelines often lean on controlled environments and expert human annotation. For instance, in creating a conversational corpus, Rohola Zandie et al. utilize professional studio recordings followed by precise digital post-processing like silence trimming and volume normalization in RyanSpeech: A Corpus for Conversational Text-to-Speech Synthesis.

To capture emotional dynamics, Mudassir Shabbir couples professional acting with human quality assessment in SEMOUR: A Scripted Emotional Speech Repository for Urdu. For indigenous languages with complex sound modifications, John Ingram et al. combine studio recordings with meticulous acoustic segmentation to study language-specific phonological rules in Connected speech processes in Warlpiri.

Whether fully automated or highly curated, the underlying insight remains the same: robust speech technology requires treating audio not as a flat text stream, but as a multi-dimensional signal that must be systematically filtered, segmented, and layered.

Key Foundational Work

BNSI Slovenian broadcast news database - speech and text corpus (2005) — contributed 1 concept elements to this direction.
Development of multimodal resources for multilingual information retrieval in the basque context (2007) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_corpus_creation_pipeline_57a926e8,
  title = {corpus creation pipeline},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/corpus-creation-pipeline/rdr_57a926e8f75a4e82b6e8b83abef691c0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_corpus_creation_pipeline_57a926e8_v1,
  title = {corpus creation pipeline},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/corpus-creation-pipeline/rdr_57a926e8f75a4e82b6e8b83abef691c0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake