NeMo Toolbox for Speech Dataset Construction

Evelina Bakhturina, Vitaly Lavrukhin, Boris Ginsburg

Abstract Automatic Speech Recognition and Text-to-Speech systems are primarily trained in a supervised fashion and require high-quality, accurately labeled speech datasets. In this work, we examine common problems with speech data and introduce a toolbox for the construction and interactive error analysis of speech datasets. The construction tool is based on Kürzinger et al. work, and, to the best of our knowledge, the dataset exploration tool is the world's first open-source tool of this kind. We demonstrate how to apply these tools to create a Russian speech dataset and analyze existing speech datasets (Multilingual LibriSpeech, Mozilla Common Voice). The tools are open sourced as a part of the NeMo framework.

Introduction

Modern Automatic Speech Recognition (ASR) models are increasingly data-hungry. As training sets scale to tens of thousands of hours, creating and maintaining high-quality labeled data becomes a significant bottleneck. Researchers typically face a choice between recording short, expensive utterances or using forced alignment to segment long-form audio like audiobooks. However, traditional alignment tools often require complex phonetic lexicons, and manual inspection of massive datasets for errors—such as mid-word cutoffs or transcript mismatches—is practically impossible.

The NeMo Toolbox

To address these challenges, we introduce an open-source toolbox within the NVIDIA NeMo framework designed to automate the construction and interactive analysis of speech datasets.

CTC-Segmentation Tool

The first component is a CTC-Segmentation tool that aligns long audio files with corresponding text using a pre-trained ASR model. By utilizing Connectionist Temporal Classification (CTC) log probabilities, the tool identifies the most likely alignment path without requiring a phonetic lexicon. Our implementation extends prior work by adding parallel processing support and end-to-end scripts that streamline the journey from raw data to training-ready clips.

The NeMo speech dataset construction and analysis workflow.

Speech Data Explorer (SDE)

The second component, the Speech Data Explorer (SDE), is a web-based application built on Plotly Dash for interactive error analysis. SDE provides a comprehensive suite of visualizations, including global statistics, duration histograms, and word accuracy distributions. It allows researchers to drill down into individual utterances to inspect waveforms and spectrograms, facilitating the identification of "silent" killers like abrupt audio endings or normalization errors.

The Speech Data Explorer (SDE) interface provides interactive visualizations for global statistics, vocabulary analysis, and individual utterance inspection.

Experimental Evaluation

We demonstrated the toolbox's utility across several datasets:

  • Russian LibriSpeech (RuLS): We constructed a new 98-hour dataset from public domain LibriVox audiobooks. By employing a K-fold re-alignment strategy—iteratively fine-tuning a model on newly aligned data to improve subsequent segmentation—we reduced the average book Word Error Rate (WER) from 45.8% to 18.83%.
  • Mozilla Common Voice (MCV): SDE was used to clean crowdsourced data. In the English subset, we identified a bloated alphabet of 149 characters. Guided by SDE, we normalized this to a standard 28-character set, significantly improving training stability.
  • Multilingual LibriSpeech (MLS): We analyzed large-scale English data to identify and filter segmentation errors.

Key Results and Insights

The application of these tools yielded tangible performance gains. Training on a combination of MCV Russian and the new RuLS dataset reduced the WER on the MCV dev set from 12.63% to 9.65%. Furthermore, SDE-guided filtering on MCV English allowed us to remove 37 hours of low-quality data, resulting in a cleaner, more effective training set.

A critical insight from this work is that interactive visualization often reveals issues that automated metrics like WER miss. For instance, visualizing the mean absolute signal value at the end of clips helped us identify "abrupt endings" where audio was prematurely truncated.

Limitations and Future Work

Currently, SDE is optimized for short ASR utterances; loading very long recordings can cause performance lag. The RuLS dataset also exhibits a gender imbalance due to the available source material. Future efforts will focus on improving speaker diversity, optimizing SDE for longer audio, and automating text normalization for a wider range of languages.

Summary updated

Abstract

Automatic Speech Recognition and Text-to-Speech systems are primarily trained in a supervised fashion and require high-quality, accurately labeled speech datasets. In this work, we examine common problems with speech data and introduce a toolbox for the construction and interactive error analysis of speech datasets. The construction tool is based on K\"urzinger et al. work, and, to the best of our knowledge, the dataset exploration tool is the world's first open-source tool of this kind. We demonstrate how to apply these tools to create a Russian speech dataset and analyze existing speech datasets (Multilingual LibriSpeech, Mozilla Common Voice). The tools are open sourced as a part of the NeMo framework.
BibTeX
@inproceedings{bakhturina2021nemo,
  author = {Evelina Bakhturina and Vitaly Lavrukhin and Boris Ginsburg},
  title = {{NeMo Toolbox for Speech Dataset Construction}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=8cggYDN0bd},
}
Figure