Data collection and performance evaluation of spoken dialogue systems: the MIT experience

James R. Glass, Joseph Polifroni, Stephanie Seneff, Victor ZueInterspeech 2000 · DOI

TL;DR

This paper introduces novel dialogue-level metrics, "query density" and "concept efficiency," to better evaluate spoken dialogue systems beyond individual utterance performance. They also detail their GALAXY infrastructure for robust off-line evaluation, crucial for iterating on complex systems like their air-travel information bot.

What Problem Does This Paper Address?

Developing effective spoken dialogue systems (like voice assistants) is a classic "chicken and the egg" problem: you need lots of real-world data to build a good system, but you need a good system to collect that data. Furthermore, evaluating these systems has traditionally focused on individual components (like speech recognition accuracy) or single turns, which doesn't capture how well the entire conversation flows. This paper tackles how to collect realistic data and, more importantly, how to measure the overall conversational performance of a system, moving beyond simple utterance-level metrics to understand the user's experience throughout a dialogue.

Methods

The authors introduce two key dialogue-level metrics:

  • Query Density (QD): Measures the average number of new concepts a user introduces per query. A higher QD suggests the user can convey more information efficiently in a single turn.
  • Concept Efficiency (CE): Measures the average number of turns it takes for a concept to be successfully understood by the system. A higher CE (closer to 1) indicates the system understands concepts quickly, while a lower CE suggests repeated turns or misunderstandings.

They also describe their GALAXY client-server architecture, which allows for flexible routing of human language technology components. For off-line evaluation, they've developed "Batchmode" and "Evaluate" servers. The Batchmode server simulates system behavior using pre-recorded data or processed inputs, while the Evaluate server specifically gathers performance statistics. Crucially, their evaluation infrastructure synchronizes dialogue context during off-line log-file reprocessing, ensuring a more realistic evaluation of system performance over time.

Datasets

  • JUPITER: A weather information domain system that collected approximately 570,000 queries from 87,000 calls over three years. This dataset was instrumental in improving speech recognition and understanding components.
  • MERCURY: A new air-travel information domain system. The paper reports experiments on a subset of 44 calls (648 utterances) made in July 2000. This system is still in its early development stages.

Key Insights and Ideas

  • Real-world data is king: The paper emphasizes that data collected under actual deployment conditions (e.g., via toll-free numbers) is far more valuable than laboratory-collected data. They observed a significant increase in word error rate (WER) when their JUPITER system first went public, highlighting the challenges of real-world acoustics and user behavior.
  • Dialogue-level metrics are essential: Traditional utterance-based metrics like WER and CER don't capture the full picture of a dialogue system's effectiveness. QD and CE provide a more holistic view of how well a user can communicate and how efficiently the system can understand information over an entire conversation.
  • Iterative development is key: The authors advocate for an iterative approach where systems are made available to real users early in the development cycle to gather data and identify weaknesses. This contrasts with developing technologies in isolated, "concocted" scenarios.
  • Infrastructure matters: A robust evaluation infrastructure like GALAXY is critical for systematically testing improvements and understanding the impact of changes on complex, multi-component dialogue systems.

Key Results

  • The JUPITER system's WER tripled when it first went public but decreased over time as more data was collected and used for model improvement.
  • For the MERCURY test set (44 calls, 648 utterances), the overall WER was 21.5%, CER was 24.4%, and UER was 42.3%.
  • The average query density (QD) for the MERCURY data was 1.47, and the concept efficiency (CE) was 0.92.
  • The MERCURY system, despite being in early development, elicited an average of 5.3 words per utterance, similar to the JUPITER system.

Limitations and Future Work

  • Limitations: The authors acknowledge that the CER and UER metrics were not available for "No Ref" utterances (where the reference orthography couldn't be parsed). They also note that guaranteeing identical system responses during off-line evaluation is virtually impossible due to dynamic data sources, potentially leading to incoherent dialogues. Handling time expressions in dynamic, time-sensitive domains like MERCURY requires specific strategies (e.g., shifting dates to the future).
  • Future Work: The authors plan to integrate their QD and CE metrics into regular evaluations. They also intend to correlate these metrics with user satisfaction using frameworks like PARADISE and by collecting user questionnaires, aiming to better quantify the impact of system performance on user experience. They also suggest that QD and CE could be used for cross-system comparisons if a common concept inventory is defined.

Summary updated

BibTeX
@inproceedings{glass2000data,
  author = {James R. Glass and Joseph Polifroni and Stephanie Seneff and Victor Zue},
  title = {{Data collection and performance evaluation of spoken dialogue systems: the MIT experience}},
  booktitle = {Interspeech},
  year = {2000},
  doi = {10.21437/icslp.2000-737},
}
Figure