Using adaptation to improve speech transcription alignment in noisy and reverberant environments

Yoshitaka Mamiya, Adriana Stan, Junichi Yamagishi, Peter Bell, Oliver Watts, Robert A. J. Clark, Simon KingSSW 2013 · OpenReview

TL;DR

This paper tackles the challenge of accurately aligning speech with text when dealing with noisy and reverberant audio, common in internet-sourced speech data. They demonstrate that adaptation techniques, specifically MAP for segmentation and CMLLR for acoustic modeling, significantly improve the amount of reliably aligned speech, with CMLLR yielding a notable 20% increase in aligned data.

What Problem Does This Paper Address?

Creating large, high-quality speech datasets for training speech synthesis and recognition systems is crucial but challenging, especially for less-resourced languages or specific domains. While manually transcribing and aligning audio is ideal, it's time-consuming and expensive. Researchers often resort to using internet-sourced audio like audiobooks, which suffer from highly variable recording conditions (noise, reverberation) that degrade the performance of automatic speech and text alignment systems. Existing alignment methods often rely on expert knowledge or expensive resources, limiting their applicability. This paper aims to improve the robustness of existing lightly supervised alignment methods to these real-world audio imperfections by employing adaptation techniques.

Methods

The authors adapt two key components of their previously developed lightly supervised speech and text alignment system:

  • Speech Segmentation (VAD): They use Maximum A Posteriori (MAP) adaptation to fine-tune the Gaussian Mixture Models (GMMs) used for Voice Activity Detection (VAD) and sentence-level segmentation. The adaptation process involves an initial decoding of the data, followed by a "data selection" step that uses a confidence measure (log likelihood ratio) to identify reliably segmented speech and silence parts. These confident segments are then used to adapt the GMMs. They also investigate the impact of this data selection step.
  • Speech Transcription Alignment: For the alignment stage, which uses grapheme-level Hidden Markov Models (HMMs), they employ Constrained Maximum Likelihood Linear Regression (CMLLR). CMLLR estimates linear transformations to adapt the acoustic models to the specific acoustic conditions of the noisy data. This is particularly useful when dealing with imperfect transcripts, as CMLLR is robust to such inaccuracies.

Datasets

  • Training/Adaptation Data: The primary dataset used is an audiobook ("A Tramp Abroad" by Mark Twain) from Librivox. For initial training of the VAD and alignment models, a small, manually labeled 10-minute speech segment was used. For adaptation, segments from the audiobook itself were used, processed chapter-by-chapter or in small groups.
  • Evaluation Data: Two chapters from the "A Tramp Abroad" audiobook were artificially degraded by adding:
  • Reverberation (using an impulse response from a domestic living room).
  • Background noise: 8-talker babble or white noise.
  • Various Signal-to-Noise Ratios (SNRs) from 10dB to 40dB.
  • Combinations of reverberation with babble or white noise. This resulted in 29 distinct testing scenarios.

  • Benchmark: The paper builds upon their previous work and the TUNDRA corpus, which is a collection of multilingual audiobooks created using their lightly supervised methods.

Key Insights and Ideas

  • Adaptation is Crucial for Noisy Data: The core insight is that standard, clean-data trained models perform poorly on real-world noisy and reverberant speech. Adaptation techniques are essential to bridge this gap.
  • Data Selection Matters for VAD Adaptation: For MAP adaptation of the VAD GMMs, simply adapting on all initially decoded data can actually reduce the discriminative power of the models. Selecting only "confident" segments for adaptation is critical for improving performance.
  • CMLLR is Highly Effective for Alignment: CMLLR adaptation of acoustic models significantly boosts the amount of reliably aligned speech, even when the initial transcripts are of low quality due to noise. This is a key finding for practical data harvesting.
  • Noise Type and SNR Impact VAD Significantly: The performance of the MAP-adapted VAD is highly sensitive to the type of noise and the SNR. Some conditions, like reverberation combined with white noise, prove particularly challenging.
  • Low WER Despite High SER: In some noisy scenarios, the Sentence Error Rate (SER) can be high, but the Word Error Rate (WER) remains relatively low. This indicates that while the overall sentence segmentation might be off, the word-level alignment within those segments can still be good, making adaptation beneficial.

Key Results

  • VAD Adaptation (MAP):
    • MAP adaptation with data selection showed an increase in the distance between average LLRs by 11% compared to MAP adaptation without data selection.
  • The CORR measure (a metric for VAD performance) showed variable improvements depending on noise type and SNR. White noise saw good improvements at high SNRs. Babble noise showed advantages at mid-range SNRs. Reverberation alone performed better with adaptation, but reverberation + white noise was problematic.
  • Alignment Adaptation (CMLLR):
    • CMLLR adaptation of acoustic models resulted in an average 20% increase in the percentage of confidently aligned data across the majority of tested scenarios.
  • This adaptation also lowered the SER of the aligned data.
  • In some cases, adaptation led to substantial improvements, e.g., a 50% improvement in SER and 20% in WER for white noise at 20dB SNR.
  • The maximum increase in confident data percentage was 62% for reverberation and white noise at 25dB SNR.

Limitations and Future Work

  • Limitations:
    • The VAD adaptation performance was highly dependent on the noise type and SNR, with reverberation and white noise proving particularly difficult.
  • The threshold for data selection in VAD adaptation was set based on one specific condition (35dB babble noise) and might not be optimal for all environments.
  • The evaluation of CMLLR adaptation was performed on a subset of the audiobook, not the entire corpus, so direct comparison to previous work on full corpora is limited.
  • Future Work:
    • Evaluate the combined effect of adaptation on both segmentation and alignment on the confident data percentage.
  • Explore cluster-based adaptation, grouping data by recording environments rather than just chapters.
  • Investigate the impact of VAD indices (CORR, FEC, MSC, OVER, NDS) on Text-to-Speech (TTS) system quality when using adapted models under various noise conditions.
  • Develop alternative methods for dealing with challenging scenarios like white noise plus reverberation.

Summary updated

Abstract

When using data retrieved from the internet to create new speech databases, the recording conditions can often be highly variable within and between sessions. This variance influences the overall performance of any automatic speech and text alignment techniques used to process this data. In this paper we discuss the use of speaker adaptation methods to address this issue. Starting from a baseline system for automatic sentence-level segmentation and speech and text alignment based on GMMs and grapheme HMMs, respectively, we employ Maximum A Posteriori (MAP) and Constrained Maximum Likelihood Linear Regression (CMLLR) techniques to model the variation in the data in order to increase the amount of confidently aligned speech. We tested 29 different scenarios, which include reverberation, 8 talker babble noise and white noise, each in various combinations and SNRs. Results show that the MAP-based segmentation’s performance is very much influenced by the noise type, as well as the presence or absence of reverberation. On the other hand, the CMLLR adaptation of the acoustic models gives an average 20% increase in the aligned data percentage for the majority of the studied scenarios.
BibTeX
@inproceedings{mamiya2013using,
  author = {Yoshitaka Mamiya and Adriana Stan and Junichi Yamagishi and Peter Bell and Oliver Watts and Robert A. J. Clark and Simon King},
  title = {{Using adaptation to improve speech transcription alignment in noisy and reverberant environments}},
  booktitle = {SSW},
  year = {2013},
  url = {https://openreview.net/forum?id=ABdKpgd4Vj},
}
Figure