HITIQA: A Data Driven Approach to Interactive Analytical Question Answering

Sharon Small, Tomek StrzalkowskiNAACL 2004

TL;DR

This paper introduces HITIQA, an interactive question answering system designed for complex, analytical queries that go beyond simple fact retrieval. HITIQA uses a novel "text framing" approach and a clarification dialogue to help users explore information spaces and refine their understanding, proving effective in evaluations with intelligence analysts.

What Problem Does This Paper Address?

Traditional question answering (QA) systems excel at "factoid" questions, where the answer type (e.g., person, country) is predictable. However, many real-world analytical questions are exploratory and underspecified, making it difficult to anticipate the answer's nature. For instance, asking "What has been Russia's reaction to U.S. bombing of Kosovo?" doesn't have a single, easily defined answer type. These analytical questions require a system that can help the user navigate a broad information space, understand the nuances of available data, and iteratively refine their query. HITIQA aims to bridge this gap by enabling analysts to pose such complex questions and receive relevant information units to build detailed reports.

Methods

HITIQA employs a two-pronged approach:

  • Text Framing: Instead of trying to fully understand a passage, HITIQA imposes a partial structure using "frames." These frames represent events or relationships and capture key entities (like LOCATION, PERSON, ORGANIZATION) and a central TOPIC. This framing process helps systematically compare different text passages and the user's question.
  • Clarification Dialogue: When a user poses a question, HITIQA first generates a "Goal frame." It then compares this to frames derived from retrieved documents. If there are mismatches (conflicts) or potential related information, HITIQA initiates an interactive dialogue. This dialogue allows the system to ask clarifying questions, helping the user refine the scope and intent of their query, and thus guiding the system towards a more precise answer. HITIQA uses specialized "typed frames" (e.g., Transfer, Relation, Property) to represent common event structures and facilitate this dialogue.

Datasets

The paper mentions the use of the WMD (Weapons of Mass Destruction) domain for adaptation and evaluation. It also refers to the TREC QA (Text REtrieval Conference Question Answering) evaluations as a context for factoid QA systems. No new benchmark datasets were introduced.

Key Insights and Ideas

  • Analytical QA is fundamentally different from Factoid QA: The inherent ambiguity and exploratory nature of analytical questions necessitate a departure from rigid answer-type prediction.
  • Interactive Dialogue is Crucial: For complex queries, a system that can engage in a clarification dialogue is essential for helping users navigate information and refine their understanding.
  • Framing as a Bridge: Text framing provides a structured yet flexible way to represent information and compare it against user queries, enabling systematic analysis and dialogue.
  • User-Centric Refinement: HITIQA empowers the user to guide the information retrieval process, making the system more adaptable to individual needs and evolving understanding.

Key Results

The paper reports on qualitative evaluations with intelligence analysts. These evaluations showed that:

  • HITIQA's approach was strongly validated by the analysts.
  • Analysts rated HITIQA's usability and usefulness highly, with a mean score of 3.73 out of 5.
  • The quality of answers generated by HITIQA was also rated well, with a mean score of 3.92 out of 5.
  • HITIQA performed particularly well when compared to current analytic tools.

Limitations and Future Work

  • Scalability of Frame Acquisition: The authors acknowledge that effective frame acquisition methods are an ongoing area of work.
  • Prototype Limitations: While the overall approach was validated, the current prototype has exposed limitations that need further development.
  • Future Work: The paper suggests further development of the interface and exploration of more advanced dialogue strategies. The authors also note that a more complete discussion of the analyst workshops and evaluation results is omitted due to space limitations.

Summary updated

BibTeX
@inproceedings{small2004hitiqa,
  author = {Sharon Small and Tomek Strzalkowski},
  title = {{HITIQA: A Data Driven Approach to Interactive Analytical Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure