Introspection and Adaptable Model Integration for Dialogue-based Question Answering

Daniel SonntagIJCAI 2009

Introduction

Modern Question Answering (QA) systems often function as "black boxes." Users submit a query and are met with either an immediate answer or a prolonged, frustrating silence that may eventually result in a "No Answer" response. This lack of transparency is particularly acute in systems that aggregate data from multiple heterogeneous sources—ranging from high-speed local databases to unpredictable web-based scrapers.

This paper addresses the Information Integration Problem, where varying response times, reliability levels, and data qualities across backend services lead to unnatural dialogue flows. To solve this, the authors introduce an "introspective" mechanism that allows the system to predict its own performance and manage user expectations through proactive meta-dialogue.

System Architecture

The proposed framework integrates several specialized components under a central QA Dialogue Manager. These include open-domain QA engines, web service composition modules, and knowledge servers.

The multi-source architecture of the dialogue-based QA system, showing the integration of open and closed domain sources.

By monitoring these diverse backends, the system can identify when a specific service is likely to be slow or unresponsive, allowing the Dialogue Manager to intervene before the user experiences a significant lag.

The Introspective Mechanism

The core innovation is the system's ability to monitor its own Information State (IS) in real-time. This introspection involves three key steps:

  1. Feature Extraction: The system tracks a wide array of features, including ASR/NLU confidence scores, query complexity, the speed of incoming data streams, and historical success rates.
  2. Trigger Events: As shown in the internal state model, specific "trigger events" (such as the completion of a semantic query) prompt the system to capture a snapshot of these features.
  3. Predictive Modeling: The system employs standard machine learning classifiers (Naive Bayes, SVM, k-NN, and C4.5) to predict the probability of a successful answer and the expected latency.
Internal Information State (IS) and trigger event mechanism used to extract features for performance prediction.

Based on these predictions, the system generates "meta-dialogue." For instance, if a 50-second search is predicted, the system might ask, "This search may take a minute; would you like me to continue?"

Evaluation and Key Results

The approach was evaluated using real user interactions across both a specific "Football Domain" (250 requests) and a general "Open Domain" (88 requests).

Predictive Performance

The researchers found that even simple models like Naive Bayes were highly effective. The system demonstrated remarkable data efficiency, learning to predict "non-accessibility" (empty results) with over 90% accuracy using only 20% of the available training data. ROC analysis confirmed that the models could reliably distinguish between successful and failing queries well before processing concluded.

User Impact

Qualitative evaluations by human judges using Likert scales revealed that introspection significantly improves the user experience:

  • Perceived Speed: Providing feedback made long response times (over 15 seconds) feel much shorter to users.
  • Reliability: Users rated the system higher for the helpfulness of its error messages and its ability to lead them to information quickly.
  • Significance: These improvements in user perception were statistically significant at $\alpha = 0.05$.

Discussion and Future Work

A key insight from this research is that transparency about system limitations increases perceived helpfulness. By using self-supervision—comparing actual results against predictions—the system can adapt to changing environments, such as temporary web service outages, without manual intervention.

Future work will focus on scaling these methods to larger dialogue corpora and moving from offline training to fully real-time, on-the-fly adaptation. Additionally, the authors aim to move beyond templated meta-dialogue toward more natural expressions of system uncertainty and effort.

Summary updated

BibTeX
@inproceedings{sonntag2009introspection,
  author = {Daniel Sonntag},
  title = {{Introspection and Adaptable Model Integration for Dialogue-based Question Answering}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2009},
}
Figure