Annotating the Semantic Consistency of Speech Recognition Hypotheses

Iryna Gurevych, Robert Porzel, Michael StrubeSIGDIAL Workshop 2002 · DOI

TL;DR

This paper tackles the challenge of ensuring speech recognition hypotheses are semantically consistent and relevant to the current domain in multi-domain dialogue systems. They introduce an annotation scheme and investigate human annotator reliability, finding that simplifying the consistency classes significantly improves agreement.

What Problem Does This Paper Address?

Modern conversational AI systems, like the SmartKom project, aim to understand and respond in natural language across various domains (e.g., cinema, home control). A crucial step is accurately interpreting speech recognition hypotheses (SRHs). However, SRHs can be noisy and may not always make sense semantically or fit the current conversational context. This paper addresses the need for a reliable way to assess the semantic consistency of SRHs and determine their relevance to specific domains, which is essential for building robust and context-aware dialogue systems.

Methods

The authors propose an annotation scheme to evaluate SRHs based on semantic consistency and domain specificity. For semantic consistency, they define three classes: consistent, semi-consistent, and inconsistent. This classification relies on mapping words in the SRH to concepts and checking if these concepts are semantically related. For domain detection, they consider a predefined set of domains relevant to the SmartKom project. A key aspect of their methodology is the investigation of human annotator reliability, using the Kappa statistic to measure agreement. They also explore revisions to their annotation scheme to improve this reliability.

Datasets

The paper describes the creation of a new corpus for their study. This corpus consists of approximately 2300 SRHs derived from audio recordings of 29 subjects participating in 8 dialogues. The data was collected using a "hidden operator test" simulating the SmartKom system. The SRHs were generated by a speech recognizer, and the resulting N-best lists were preprocessed into annotation files.

Key Insights and Ideas

  • Semantic Consistency is Key: The paper highlights that semantic consistency is a fundamental requirement for dialogue systems to function correctly, especially when dealing with imperfect speech recognition.
  • Human Agreement is Challenging: Achieving high agreement among human annotators for semantic consistency proved difficult with the initial, more granular annotation scheme.
  • Simplification Improves Reliability: Merging the "semi-consistent" class with either "consistent" or "inconsistent" significantly improved inter-annotator agreement, suggesting that simpler classification schemes can be more practical.
  • Domain Detection Depends on Consistency: The authors posit that domain detection is best performed on SRHs that are already deemed semantically consistent, as inconsistent hypotheses are unlikely to be relevant to any domain.
  • Context is Crucial (but excluded here): The study intentionally excluded dialogue context to focus on the inherent consistency of individual SRHs, acknowledging that context would be vital in a real system.

Key Results

The initial annotation scheme for semantic consistency yielded a Kappa statistic of 0.58, indicating only moderate agreement among annotators. When the "semi-consistent" class was merged with "inconsistent," the Kappa statistic improved to 0.7, suggesting a considerable increase in reliability. Merging "semi-consistent" with "consistent" did not yield a significant improvement (Kappa = 0.59). The distribution of SRHs across consistency classes showed a majority (51%) were classified as "consistent," with a substantial portion (38.7%) as "inconsistent."

Limitations and Future Work

The authors acknowledge that the relatively low agreement in their initial experiment could be due to the small number of annotators and the fine-grained distinction between consistency classes. They also note that annotators sometimes struggled to ignore functional words as per the scheme's instructions. Future work includes integrating these findings into the SmartKom domain modeling component and conducting further experiments. They also suggest that future human annotation studies could focus on automatically generated conceptual mappings rather than raw SRHs, as machines can handle word-to-concept mapping more directly. The reliability of domain assignments was not the focus of this paper and will be published separately.

Summary updated

BibTeX
@inproceedings{gurevych2002annotating,
  author = {Iryna Gurevych and Robert Porzel and Michael Strube},
  title = {{Annotating the Semantic Consistency of Speech Recognition Hypotheses}},
  booktitle = {SIGDIAL Workshop},
  year = {2002},
  doi = {10.3115/1118121.1118128},
}
Figure