A Comprehensive Evaluation of Incremental Speech Recognition and Diarization for Conversational AI

Angus Addlesee, Yanchao Yu, Arash EshghiCOLING 2020

TL;DR

This paper provides a comprehensive evaluation of leading incremental Automatic Speech Recognition (ASR) and Speaker Diarization (SD) systems for conversational AI. While current systems show impressive overall accuracy, none are yet fully capable of handling the complexities of natural, spontaneous, real-time conversations, particularly regarding disfluencies and speech overlaps. Microsoft leads in incremental ASR, while IBM excels in incremental SD and robustness to overlaps.

What Problem Does This Paper Address?

Conversational AI systems, like chatbots or virtual assistants, need to understand and respond to human speech in real-time. This requires not just accurate transcription (ASR) but also the ability to distinguish between different speakers (Speaker Diarization or SD), especially in multi-party conversations. Traditional ASR systems often process entire utterances, which is too slow for natural dialogue. Incremental ASR aims to address this by producing hypotheses word-by-word with minimal delay. However, real-world conversations are messy: they contain disfluencies (like "uhm," "sorry," self-corrections) and speech overlaps (when people talk at the same time). Evaluating how well current commercial ASR and SD services handle these challenges in an incremental fashion is crucial for building truly natural and responsive conversational agents.

Methods

The authors evaluate three leading commercial ASR services (Microsoft Azure, IBM Watson, and Google Cloud) and two SD services (Google Cloud and IBM Watson). For ASR, they focus on incremental performance metrics beyond the standard Word Error Rate (WER), including:

  • Latency: How quickly hypotheses are produced (First Occurrence latency) and how quickly a final decision is made for a word (Final Decision latency).
  • Stability: How likely hypotheses are to change over time (measured by Word Survival Rate).
  • Fidelity to Disfluencies: Whether the ASR system preserves or "cleans" disfluent speech (like filled pauses and edit terms) and self-corrections. This is measured by changes in WER when disfluent material is removed from the ground truth.
  • Robustness to Speech Overlaps: How well the ASR performs when multiple speakers talk simultaneously. This is measured by the difference in WER between merged audio streams and individual speaker channels.

For Speaker Diarization, they use the Diarization Error Rate (DER), which includes speaker error, false alarm speech, and missed speech rates.

Datasets

  • Switchboard Corpus: A large corpus of dyadic telephone conversations used for evaluating ASR performance, including incremental metrics, disfluency fidelity, and robustness to overlaps. This corpus is chosen for its open-domain nature and existing annotations for disfluencies.
  • AVDIAR Corpus: A multi-party audio-visual diarization dataset with up to four speakers, used to evaluate SD systems in more complex conversational settings.

Key Insights and Ideas

  • Incremental vs. Non-Incremental Performance Gap: Incremental ASR systems have significantly higher WERs than their non-incremental (bidirectional) counterparts. This highlights the inherent difficulty of processing speech without future context.
  • Disfluency Preservation is Key for Naturalness: The authors argue that preserving disfluent material is beneficial for downstream conversational AI components, allowing them to better understand human intent and conversational flow.
  • Trade-offs Exist: No single system excels at all aspects. Microsoft is strong in incremental ASR responsiveness and stability, while IBM leads in incremental SD and robustness to speech overlaps. Google offers a balance but doesn't lead in any specific area.
  • Real-time SD is Challenging: Even for multi-party conversations, current SD systems struggle with audio quality variations and the sheer complexity of distinguishing speakers accurately in real-time.

Key Results

  • Incremental ASR Performance:
    • Overall WER: Microsoft (32.89%), Google (33.62%), IBM (35.55%). Microsoft performs best.
  • Latency: Microsoft is the most responsive, hypothesizing words very quickly after they are spoken. IBM is less responsive, with more varied latency.
  • Stability: IBM and Microsoft show high stability, with IBM being particularly stable due to its less frequent hypothesis updates.
  • Disfluency Fidelity: Microsoft preserves filled pauses, leading to a WER increase when they are filtered. IBM and Google tend to filter filled pauses, resulting in a WER decrease. All systems struggle with incremental self-correction processing.
  • Robustness to Overlaps: IBM is the most robust to speech overlaps, showing the smallest WER improvement when overlaps are removed. Microsoft and Google are less robust.
  • Speaker Diarization Performance:
    • Switchboard (Dyadic): IBM (15.33% DER) significantly outperforms Google (43.93% DER), largely due to Google's higher speaker error rate.
  • AVDIAR (Multi-party): IBM (48.94% DER overall) consistently outperforms Google (68.79% DER overall) across different numbers of speakers. IBM also shows better performance on dyadic conversations within AVDIAR.

Limitations and Future Work

  • System Architectures Undisclosed: The underlying architectures of the commercial ASR and SD systems are not public, limiting the ability to tie specific metric findings to architectural choices.
  • Audio Channel Bleeding: In the Switchboard corpus, some audio bleeding between channels affects the isolated speaker WERs, though the authors state this doesn't impact their findings as the relative performance is still valid.
  • AVDIAR Quality: The lower audio quality in AVDIAR negatively impacts both systems' performance, particularly for Google, which struggles to recognize speakers in some streams.
  • Future Work: The authors plan to release their evaluation framework and Switchboard transcripts to encourage further research and the development of systems that meet these incremental and disfluency-aware metrics. They emphasize that none of the current systems are yet adequate for reliably handling natural spontaneous conversations in real-time.

Summary updated

BibTeX
@inproceedings{addlesee2020comprehensive,
  author = {Angus Addlesee and Yanchao Yu and Arash Eshghi},
  title = {{A Comprehensive Evaluation of Incremental Speech Recognition and Diarization for Conversational AI}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
}
Figure