Automatic concept identification in goal-oriented conversations

Ananlada Chotimongkol, Alexander I. RudnickyInterspeech 2002 · OpenReview

TL;DR

This paper tackles the challenge of automatically discovering key concepts within goal-oriented human conversations without pre-existing labels. They introduce a novel evaluation metric that accounts for concepts being represented by multiple clusters, demonstrating that a Kullback-Liebler distance-based clustering approach outperforms mutual information-based clustering for this task.

What Problem Does This Paper Address?

Building useful spoken language systems often requires understanding the domain of conversation. Manually annotating large corpora to identify these domain concepts is time-consuming and expensive. This paper aims to automate this process by identifying "key domain concepts" from unannotated, goal-oriented human conversations. The authors focus on goal-oriented dialogues because they exhibit a clear structure that can be leveraged for concept identification. This is a crucial first step towards automatically inferring domain ontologies, which can then inform the development of more intelligent conversational agents.

Methods

The core of the paper lies in comparing two unsupervised word-clustering algorithms: one based on mutual information (MI) and another based on Kullback-Liebler (KL) distance. Both algorithms are hierarchical, iteratively merging words or clusters based on their statistical similarity with surrounding words.

  • Mutual Information (MI)-based clustering: This method defines similarity based on the mutual information between adjacent words or clusters. It greedily merges clusters to minimize the loss in average mutual information.
  • Kullback-Liebler (KL)-based clustering: This approach uses the KL distance to measure similarity. To handle potential zero probabilities, they employ a symmetric Jensen-Shannon divergence variant. The similarity between clusters is determined by the KL distance between their left and right context probabilities.

A key methodological contribution is the introduction of a new evaluation metric called "singularity score" (SS). Traditional metrics like precision and recall, borrowed from topic detection, are insufficient because their work allows multiple clusters to represent a single concept. The singularity score penalizes concepts that are split across many clusters, thus better capturing the quality of the clustering for concept identification. They also propose an automatic stopping criterion for the clustering process based on the growth of log(AMIdelta) (the difference in average mutual information between successive iterations).

Datasets

The experiments were conducted on the CMU Travel Agent corpus. This corpus consists of 39 goal-oriented human-human dialogues in the air travel domain, totaling 2,196 utterances. The vocabulary size is 950 words. The authors define their reference concepts based on language modeling classes from the CMU Communicator system, resulting in 15 domain concepts with 188 concept words. Words not belonging to any domain concept are grouped into a single "general concept."

Key Insights and Ideas

  • KL-based clustering is superior for concept identification: The KL-based approach consistently outperformed the MI-based approach, both in terms of optimal cluster quality and quality achieved at an automatic stopping point.
  • The importance of a concept-aware evaluation metric: The proposed "singularity score" proved more insightful than traditional metrics like F1-score. It effectively penalized the splitting of concepts into multiple clusters, which is a crucial aspect of identifying coherent conceptual units.
  • Faster convergence with KL-based clustering: The KL-based algorithm reached its optimal quality much faster than the MI-based method. This is attributed to its tendency to merge concept words first, rather than creating new clusters as frequently as the MI-based approach.
  • Automatic stopping criteria are practical: The paper demonstrates that an automatic stopping criterion can yield good quality clusters, which is essential since reference concepts are not available in real-world scenarios.

Key Results

  • Optimal Quality: KL-based clustering achieved a higher optimal quality score (QS) of 0.66 compared to MI-based clustering's 0.64.
  • Automatic Stopping Quality: At their respective stopping criteria, KL-based clustering achieved a QS of 0.62, while MI-based clustering achieved 0.52.
  • Precision: KL-based clustering achieved higher precision (0.82 optimal, 0.74 at stop) compared to MI-based clustering (0.77 optimal, 0.78 at stop).
  • Recall: Both methods achieved similar recall values (0.49 optimal for both, 0.49 at stop for KL and 0.34 at stop for MI).
  • Singularity Score: KL-based clustering generally maintained a higher singularity score, indicating better grouping of concept words.

Limitations and Future Work

  • Limitations: The paper acknowledges that the current approach restricts each concept member to a single word. They also note that while the KL-based clustering is faster, the MI-based clustering's slow convergence might lead to merging irrelevant words or different concepts.
  • Future Work: The authors plan to improve clustering performance by incorporating additional information from the turn structure of dialogs. They hypothesize that concepts introduced by one participant and echoed by another, as well as co-occurrence of words across turns, can augment the current bigram co-occurrence features.

Summary updated

Abstract

We address the problem of identifying key domain concepts automatically from an unannotated corpus of goal-oriented humanhuman conversations. We examine two clustering algorithms, one based on mutual information and another one based on Kullback- Liebler distance. In order to compare the results from both techniques quantitatively, we evaluate the outcome clusters against reference concept labels using precision and recall metrics adopted from the evaluation of topic identification task. However, since our system allows more than one cluster to associate with each concept an additional metric, a singularity score, is added to better capture cluster quality. Based on the proposed quality metrics, the results show that Kullback-Liebler-based clustering outperforms mutual informationbased clustering for both the optimal quality and the quality achieved using an automatic stopping criterion.
BibTeX
@inproceedings{chotimongkol2002automatic,
  author = {Ananlada Chotimongkol and Alexander I. Rudnicky},
  title = {{Automatic concept identification in goal-oriented conversations}},
  booktitle = {Interspeech},
  year = {2002},
  url = {https://openreview.net/forum?id=orp5IVB4wS},
}
Figure