Automatic Generation of Information-seeking Questions Using Concept Clusters

Shuguang Li, Suresh ManandharACL 2009

TL;DR

This paper tackles the challenge of generating clarifying questions in interactive question answering (QA) systems. They propose a novel approach using "concept clusters" derived from answer documents to identify topics that best distinguish between potential answers, leading to more effective disambiguation.

What Problem Does This Paper Address?

Traditional Question Answering (QA) systems often struggle with ambiguous or underspecified user queries. When a QA system returns multiple answers, these answers might cover different aspects or topics. To effectively satisfy a user's information need, interactive dialogue is crucial to clarify their intent. However, generating good clarifying questions is difficult. A good question should not only be relevant to the original query but also help the user distinguish between the provided answers. This paper addresses the problem of automatically generating such informative and disambiguating question topics.

Methods

The core of the proposed method lies in using concept clusters to represent the topics present in the answers.

  1. Concept Cluster Generation: They combine existing methods (Clustering by Committee, Google logs, Google Sets) to create a large set of concept clusters. Each cluster is a collection of related subtopics.
  2. Document Representation: Documents (specifically, Google snippets retrieved for each answer) are represented as vectors where each dimension corresponds to a concept cluster, and the value is a score indicating the presence and importance of that cluster's subtopics in the document. They use a tf-idf scheme for weighting subtopics.
  3. Separability Measure: To rank these concept clusters, they introduce a "separability measure." This metric quantifies how well a cluster's subtopics can distinguish between different answers. It penalizes larger clusters and rewards clusters whose subtopics are present across multiple answers and help differentiate them.
  4. Cluster Ranking Algorithm: An algorithm is presented to filter and rank concept clusters based on their separability scores, prioritizing those that are present in the answers and contribute to distinguishing them.

Datasets

  • TREC 2004-2007 Data: Used for list questions. The authors selected 200 questions with at least 3 answers from this dataset.
  • Ambiguous Question Collection: A custom collection of 200 questions manually chosen from TREC 1999-2007 data and existing literature (Hori et al., 2003; Burger et al., 2001).

Key Insights and Ideas

  • Answers Hold the Key: The most significant insight is that the answers themselves contain crucial information for generating effective clarifying questions. Instead of just analyzing the question or general document collections, focusing on how answers relate to different topics is key.
  • Distinguishing Power is Crucial: A good clarifying question topic isn't just relevant; it must have the power to separate one answer from another. This "separability" is the core metric for ranking potential topics.
  • Concept Clusters as Topics: Using concept clusters, which are collections of related subtopics, provides a richer representation of potential question topics than single keywords.

Key Results

The proposed approach significantly outperforms a baseline method that simply ranks clusters based on their frequency and distribution across answers.

  • List Questions: The authors' method (List_S) achieved a Mean Average Precision (MAP) of 60.3%, compared to 41.3% for the baseline (List_B). Precision at 1 (P@1) was 90.0% vs. 42.1%.
  • Ambiguous Questions: For ambiguous questions, their method (Ambiguous_S) achieved a MAP of 53.6%, significantly better than the baseline (Ambiguous_B) at 31.1%. P@1 was 71.1% vs. 33.2%.
  • Reduced "Bad" Clusters: The error rate for questions where all top three clusters were labeled "bad" (Err@3) was substantially lower with their method (11.0% for list questions, 29.7% for ambiguous questions) compared to the baseline.

Limitations and Future Work

  • Ignoring Question Clusters: The current approach primarily focuses on clusters derived from answer documents and largely ignores clusters associated with the original question itself.
  • Future Work: The authors plan to investigate the relationships between concept clusters found in both the question and the answers to further refine the topic generation process. They also acknowledge that unstructured web documents can sometimes contain multiple subtopics within a single answer's context, and their current method doesn't explicitly handle this nuance beyond keyword matching.

Summary updated

BibTeX
@inproceedings{li2009automatic,
  author = {Shuguang Li and Suresh Manandhar},
  title = {{Automatic Generation of Information-seeking Questions Using Concept Clusters}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2009},
}
Figure