Abstract
In this paper, we introduce Libriheavy, a large-scale ASR corpus consisting of 50,000 hours of read English speech derived from LibriVox. To the best of our knowledge, Libriheavy is the largest freely-available corpus of speech with supervisions. Different from other open-sourced datasets that only provide normalized transcriptions, Libriheavy contains richer information such as punctuation, casing and text context, which brings more flexibility for system building. Specifically, we propose a general and efficient pipeline to locate, align and segment the audios in previously published Librilight to its corresponding texts. The same as Librilight, Libriheavy also has three training subsets small, medium, large of the sizes 500h, 5000h, 50000h respectively. We also extract the dev and test evaluation sets from the aligned audios and guarantee there is no overlapping speakers and books in training sets. Baseline systems are built on the popular CTC-Attention and transducer models. Additionally, we open-source our dataset creation pipeline which can also be used to other audio alignment tasks.
Introduction
Most open-source Automatic Speech Recognition (ASR) datasets, such as the widely used LibriSpeech, provide "normalized" transcripts—text that is entirely uppercase or lowercase with no punctuation. While this simplifies the initial task for models, it does not reflect natural language usage. Consequently, developers often must implement separate post-processing models for punctuation and capitalization to make the output human-readable.
Furthermore, most datasets treat audio clips as isolated segments, ignoring the vital cross-utterance context that humans use to resolve ambiguities. Libriheavy addresses these gaps by providing a massive-scale dataset (50,000 hours) that includes full formatting and preceding text context, enabling the training of end-to-end models that produce human-ready text directly.
Methods: The Alignment Pipeline
The authors developed a sophisticated two-stage alignment pipeline to map 50,000 hours of audio from the Librilight project back to the original source texts from LibriVox.
- First Alignment Stage: A basic ASR model generates a "rough" transcript (the query). The pipeline then utilizes suffix arrays to find close matches between this query and the original book text (the target). This method is highly efficient for locating specific audio clips within massive text corpora.
- Second Alignment Stage: Once the general region is identified, Levenshtein alignment is used to achieve word-level precision.
- Audio Segmentation: Instead of arbitrary cuts, a two-stage scoring method identifies silence and sentence-ending punctuation (., ?, !), ensuring segments are linguistically meaningful.
- Context Integration: For every segment, the preceding 1,000 bytes of text are stored, allowing models to leverage historical context.
Datasets
Libriheavy is structured into three training tiers to facilitate research at different scales: Small (500h), Medium (5,000h), and Large (50,000h). The dataset also includes new dev and test sets, meticulously filtered to ensure no overlap with training speakers or books.
Key Insights and Results
A significant finding of this work is that the "formatting gap"—the performance difference between models trained on normalized vs. formatted text—shrinks as the dataset scale increases. At the 50,000-hour scale, models can learn punctuation and casing "for free" without significant loss in accuracy.
The authors evaluated the dataset using CTC-Attention (Wenet) and Transducer (Zipformer) architectures. On the Libriheavy-Large set, the Transducer model achieved a Word Error Rate (WER) of 2.20% on "clean" speech and 5.57% on "other" speech. When comparing a model trained on "Upper case No Punctuation" (UNP) vs. "Casing & Punctuation" (C&P), the C&P model performed nearly as well as the UNP model (1.72% vs 1.62% WER on LibriSpeech clean).
Conclusion
Libriheavy represents a major step forward for open-source ASR, providing the scale and metadata necessary for modern, contextualized models. While currently limited to read speech from audiobooks, the provided open-source alignment pipeline can be adapted to other audio tasks. Future work will focus on architectures that more effectively exploit the provided pre_texts metadata to improve long-form recognition.
Summary updated