Study on integration of speaker diarization with speaker adaptive speech recognition for broadcast transcription

Jan Silovsky, Petr Cerva, Jindrich Zdansky, Jan NouzaInterspeech 2012

TL;DR

This paper explores how integrating speech transcripts into speaker diarization can improve both diarization accuracy and efficiency. It also demonstrates that using the output of this improved diarization to adapt speech recognition models leads to better transcription performance compared to traditional adaptation methods.

What Problem Does This Paper Address?

Transcribing broadcast audio, especially with multiple speakers, is a challenging task. Speaker diarization (figuring out "who spoke when") and speaker adaptive speech recognition (making the speech recognition system better for a specific speaker) are crucial components. Traditionally, these systems operate somewhat independently. Speaker adaptation often relies on preliminary transcripts, and speaker diarization can be hindered by non-speech sounds. This paper investigates a tighter integration: how can speech transcripts help diarization, and how can diarization output improve speaker adaptation for more accurate broadcast transcription?

Methods

The core of the paper lies in its novel integration strategy.

  • Speaker Diarization with Transcripts: Instead of relying on traditional Speech Activity Detection (SAD) to identify speech segments, this system uses the provided speech transcripts. This means that even non-speech sounds (like breathing or hesitations) that are part of a speaker's utterance are not discarded, preventing the segmentation from being broken by these events. The system uses a two-stage clustering approach: an initial stage using Bayesian Information Criterion (BIC) and a second stage using i-vectors and cosine distance for similarity measurement.
  • Speaker Adaptation: The paper employs Constrained Maximum Likelihood Linear Regression (CMLLR) for speaker adaptation. This technique adjusts acoustic models to better match a speaker's voice. The key innovation here is using the output of the integrated diarization system to define the segments for adaptation, rather than relying on fixed-length blocks or segments identified solely by change-point detection.

Datasets

  • Training/Development: A portion of the COST278 multilingual pan-European broadcast news database was used.
  • Testing:
    • Multilingual COST278 test set: Included 5 languages (Belgian Dutch, Czech, Hungarian, Slovak, Slovenian) from 15 TV news shows, totaling 6.3 hours. Speaker counts varied widely (2-78).
  • Czech test set: Comprised 16 Czech broadcast programs (TV and radio), totaling 4.3 hours. Speaker counts ranged from 2-55. This set included interviews, leading to more recordings with fewer speakers.

Key Insights and Ideas

  • Transcripts as a Diarization Aid: Using speech transcripts significantly improves speaker diarization. By leveraging transcript information, the system can better handle non-speech events, leading to more accurate segmentation and clustering.
  • Diarization Output for Adaptation: Adapting speech recognition models using segments defined by the diarization output yields better results than adapting based on fixed-length blocks or segments identified solely by change-point detection. This highlights the value of accurate speaker segmentation for effective adaptation.
  • Efficiency Gains: Incorporating transcripts into diarization not only improves accuracy but also reduces computational cost by reducing the number of change-point candidates to consider.
  • Language Independence: The diarization system's reliance on transcript structure rather than lexical content makes it less dependent on the language of the Automatic Speech Recognition (ASR) system.

Key Results

  • Diarization Performance:
    • On the multilingual COST278 test set, using transcripts improved the F-rate for change-point detection from 66.6% to 77.2% and reduced the Diarization Error Rate (DER) from 13.9% to 11.5%.
  • Similar improvements were observed on the Czech test set, with the F-rate increasing from 64.6% to 75.7% and DER decreasing from 21.7% to 19.5%.
  • The Word Breakage (WB) rate was significantly reduced by incorporating transcript information.
  • Speech Recognition Performance (SA System):
    • Adaptation using diarization-defined clusters outperformed adaptation on fixed-length blocks and speaker-homogeneous segments.
  • For the Czech test set, the best SA system achieved a Word Error Rate (WER) of 13.12% when adapting based on diarization output, representing a 9.3% relative improvement over a baseline system.
  • The performance of automatic adaptation techniques was close to that achieved using oracle (human-annotated) transcripts.

Limitations and Future Work

  • Oracle Transcripts: While the paper shows that using oracle transcripts yields further slight improvements in segmentation, the primary focus is on the benefits of automatic transcripts.
  • Language Dependence of ASR: The paper notes that the diarization system's independence from language is an advantage, but the overall system's performance is still dependent on the accuracy of the underlying ASR system, which is language-dependent.
  • Future Work: The paper doesn't explicitly detail future work, but the demonstrated success of integrating transcripts into diarization and using diarization output for adaptation suggests further exploration of these synergistic approaches. Investigating more advanced adaptation techniques or exploring how diarization can inform other aspects of the ASR pipeline could be promising directions.

Summary updated

BibTeX
@inproceedings{silovsky2012study,
  author = {Jan Silovsky and Petr Cerva and Jindrich Zdansky and Jan Nouza},
  title = {{Study on integration of speaker diarization with speaker adaptive speech recognition for broadcast transcription}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure