Libriheavy: a 50, 000 hours ASR corpus with punctuation casing and context

Wei Kang, Xiaoyu Yang, Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, Daniel Povey

Abstract

In this paper, we introduce Libriheavy, a large-scale ASR corpus consisting of 50,000 hours of read English speech derived from LibriVox. To the best of our knowledge, Libriheavy is the largest freely-available corpus of speech with supervisions. Different from other open-sourced datasets that only provide normalized transcriptions, Libriheavy contains richer information such as punctuation, casing and text context, which brings more flexibility for system building. Specifically, we propose a general and efficient pipeline to locate, align and segment the audios in previously published Librilight to its corresponding texts. The same as Librilight, Libriheavy also has three training subsets small, medium, large of the sizes 500h, 5000h, 50000h respectively. We also extract the dev and test evaluation sets from the aligned audios and guarantee there is no overlapping speakers and books in training sets. Baseline systems are built on the popular CTC-Attention and transducer models. Additionally, we open-source our dataset creation pipeline which can also be used to other audio alignment tasks.

Introduction

Most open-source Automatic Speech Recognition (ASR) datasets, such as the widely used LibriSpeech, provide "normalized" transcripts—text that is entirely uppercase or lowercase with no punctuation. While this simplifies the initial task for models, it does not reflect natural language usage. Consequently, developers often must implement separate post-processing models for punctuation and capitalization to make the output human-readable.

Furthermore, most datasets treat audio clips as isolated segments, ignoring the vital cross-utterance context that humans use to resolve ambiguities. Libriheavy addresses these gaps by providing a massive-scale dataset (50,000 hours) that includes full formatting and preceding text context, enabling the training of end-to-end models that produce human-ready text directly.

Methods: The Alignment Pipeline

The authors developed a sophisticated two-stage alignment pipeline to map 50,000 hours of audio from the Librilight project back to the original source texts from LibriVox.

  1. First Alignment Stage: A basic ASR model generates a "rough" transcript (the query). The pipeline then utilizes suffix arrays to find close matches between this query and the original book text (the target). This method is highly efficient for locating specific audio clips within massive text corpora.
  2. Second Alignment Stage: Once the general region is identified, Levenshtein alignment is used to achieve word-level precision.
  3. Audio Segmentation: Instead of arbitrary cuts, a two-stage scoring method identifies silence and sentence-ending punctuation (., ?, !), ensuring segments are linguistically meaningful.
  4. Context Integration: For every segment, the preceding 1,000 bytes of text are stored, allowing models to leverage historical context.
Suffix array alignment process for locating audio in target text

Datasets

Libriheavy is structured into three training tiers to facilitate research at different scales: Small (500h), Medium (5,000h), and Large (50,000h). The dataset also includes new dev and test sets, meticulously filtered to ensure no overlap with training speakers or books.

Statistics of the Libriheavy dataset subsets

Key Insights and Results

A significant finding of this work is that the "formatting gap"—the performance difference between models trained on normalized vs. formatted text—shrinks as the dataset scale increases. At the 50,000-hour scale, models can learn punctuation and casing "for free" without significant loss in accuracy.

The authors evaluated the dataset using CTC-Attention (Wenet) and Transducer (Zipformer) architectures. On the Libriheavy-Large set, the Transducer model achieved a Word Error Rate (WER) of 2.20% on "clean" speech and 5.57% on "other" speech. When comparing a model trained on "Upper case No Punctuation" (UNP) vs. "Casing & Punctuation" (C&P), the C&P model performed nearly as well as the UNP model (1.72% vs 1.62% WER on LibriSpeech clean).

Conclusion

Libriheavy represents a major step forward for open-source ASR, providing the scale and metadata necessary for modern, contextualized models. While currently limited to read speech from audiobooks, the provided open-source alignment pipeline can be adapted to other audio tasks. Future work will focus on architectures that more effectively exploit the provided pre_texts metadata to improve long-form recognition.

Summary updated

Abstract

In this paper, we introduce Libriheavy, a large-scale ASR corpus consisting of 50,000 hours of read English speech derived from LibriVox. To the best of our knowledge, Libriheavy is the largest freely-available corpus of speech with supervisions. Different from other open-sourced datasets that only provide normalized transcriptions, Libriheavy contains richer information such as punctuation, casing and text context, which brings more flexibility for system building. Specifically, we propose a general and efficient pipeline to locate, align and segment the audios in previously published Librilight to its corresponding texts. The same as Librilight, Libriheavy also has three training subsets small, medium, large of the sizes 500h, 5000h, 50000h respectively. We also extract the dev and test evaluation sets from the aligned audios and guarantee there is no overlapping speakers and books in training sets. Baseline systems are built on the popular CTC-Attention and transducer models. Additionally, we open-source our dataset creatation pipeline which can also be used to other audio alignment tasks.
BibTeX
@inproceedings{kang2024libriheavy,
  author = {Wei Kang and Xiaoyu Yang and Zengwei Yao and Fangjun Kuang and Yifan Yang and Liyong Guo and Long Lin and Daniel Povey},
  title = {{Libriheavy: a 50, 000 hours ASR corpus with punctuation casing and context}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=7W5lU7gbhD},
}
Figure