Abstract
This paper introduces the concept of "introspection" to Question Answering (QA) systems—the ability for a system to evaluate its own intermediate search results before attempting to extract an answer. By using machine learning to classify search engine "hit lists" as good or bad based on score patterns and question types, the authors achieved significant improvements in system awareness, including a 25% relative improvement in accuracy for challenging definition-style questions.
Introduction
Most QA systems follow a linear pipeline: analyze the question, search a document collection, and extract an answer from the top results. However, if the search engine returns a "hit list" that does not contain the answer, the system will likely hallucinate a wrong answer or waste computational resources processing irrelevant data.
The authors address the need for a feedback loop, or introspection mechanism. A system needs to know when its search results are poor so it can decide to retry the search with a different strategy, signal that "No Answer" is the correct response, or adjust its confidence in the final output.
Methodology
The authors frame introspection as a concept-learning task: given a list of search results (a hit list), can a classifier label it as "good" (contains at least one correct answer) or "bad" (contains none)?
They utilized two primary machine learning models: C4.5 Decision Trees (DT) and Multi-layer Perceptron Neural Networks (NN). The novelty lies in their feature engineering, which focuses on the structural shape of the search results rather than the text itself:
- Score Progression (F, S, D): Instead of looking at absolute scores, they analyzed the relative drop-off between ranked results. A sharp drop-off after the top results often indicates a highly relevant match.
- Question Type (T, P): Binary features representing the expected answer type (e.g., PERSON, DATE, or DEFINITION) were included, as different question types produce different score distributions.
Experimental Setup
The study used data from the TREC-9 and TREC-10 QA-Tracks, the gold standard for QA research at the time. TREC-9 (688 examples) was used for training, while TREC-10 (498 examples) evaluated generalization to unseen data. The data was further categorized into subsets like ABBREV, DEFINE, PERSON, and PLACE to observe performance across semantic domains.
Key Results
The introspection models consistently outperformed the majority-class baseline. The most dominant feature was the relative change in term scores (F); a flat score distribution typically suggested the engine was merely guessing across mediocre matches.
As illustrated in the performance data, the system was particularly effective at introspecting "Definition" questions. The Neural Network achieved a 25.3% relative gain in accuracy over the majority baseline for definition questions on the TREC-10 dataset. For the "FULL" dataset, the Decision Tree showed a 18.7% relative improvement.
Analysis and Reliability
While the classifiers are adept at identifying "good" hit lists, identifying "bad" hit lists remains a more difficult challenge.
The breakdown of accuracy reveals that while precision for "good" lists is high (often exceeding 80%), the accuracy for "bad" lists is significantly lower. For example, in the DEFINE category, the system correctly identified about half of the "bad" lists. While this is a significant step toward preventing false answers, it highlights a remaining gap in detection. Additionally, the system struggled with the "THING" category due to inconsistent score patterns in its catch-all question set.
Conclusion
Introspection provides a promising mechanism for improving the reliability of QA systems. By identifying poor search results early, systems can avoid downstream errors. Future work will focus on end-to-end integration and incorporating semantic features—analyzing the meaning of the snippets themselves—to further improve the detection of "bad" hit lists.
Summary updated