Improving Case Retrieval by Remembering Questions

AAAI 1996

TL;DR

This paper introduces a "use-adapted" case-based reasoning (CBR) system that learns to improve its case retrieval performance over time by remembering past user interactions. Instead of relying solely on static case indexing, the system builds a history of questions asked and the relevance of retrieved cases, enabling it to adapt its retrieval strategies based on usage patterns. This approach aims to overcome the limitations of traditional CBR systems that require extensive upfront design and struggle with evolving user needs or data.

What Problem Does This Paper Address?

Building effective case-based reasoning (CBR) systems typically involves significant upfront effort in defining what constitutes a "case," structuring its content, selecting a vocabulary for indexing, and then indexing the cases themselves. This process is often complex and requires deep domain expertise. Furthermore, once deployed, the system's performance is largely fixed, making it difficult to adapt to changes in user behavior, evolving data, or new applications. The authors highlight that the "right" way to index a case might not be static but rather dependent on how users interact with the system and what they are trying to achieve. This paper addresses the challenge of making CBR systems more adaptable and user-centric after deployment, by learning from actual usage rather than relying solely on initial design decisions.

Methods

The core innovation is the concept of a use-adapted case retriever. This system augments a standard CBR retriever with a module that maintains a history of user-system interactions. Specifically, it records:

  • Questions asked by the user.
  • Cases retrieved in response.
  • User-assigned relevance markers (+, -, or 0) for each retrieved case.

This history forms a "question base" and "relevance links" between questions and cases. The paper proposes three key techniques that leverage this history:

  1. Recognizing Questions: If a new user query is similar to a previously asked question in the history, the system can leverage the known relevant cases associated with that past question to improve retrieval. This is a "short form" recognition, using the initial query.
  2. Automatic Question Expansion: When the system encounters an impasse during retrieval (e.g., a feature is needed but not provided in the query), it can use the question base to "guess" the value of the missing feature. It polls positively associated cases from similar past questions to vote on the most likely feature value. This reduces the need for users to interactively expand their queries.
  3. Pruning Cases: This technique uses negative associations between questions and cases. If a new query strongly matches queries that were previously marked as irrelevant to a particular case, that case can be removed from the retrieval set, thus reducing noise and improving precision.

Datasets

The paper describes experiments conducted using a base help-desk system with a case base of 293 cases. These cases were bug reports consisting of unstructured text. A vocabulary of approximately 348 keywords was identified. For evaluation, seven subjects wrote queries. The collected queries were split into a training set (132 queries) and a test set (107 queries). The test set was further divided into those with a relevant history (43 queries) and those without (64 queries). No new benchmarks were introduced; existing help-desk data was used.

Key Insights and Ideas

  • Usage is Key to Adaptation: The most significant insight is that a system's performance can be dramatically improved by learning from how it's actually used, rather than relying solely on initial design. The "right" way to retrieve a case is context-dependent and emerges through interaction.
  • Remembering Questions is Powerful: Storing past questions and their associated relevance judgments creates a valuable knowledge base. This allows the system to recognize recurring information needs and leverage past successes.
  • Reducing User Burden: Techniques like automatic question expansion significantly reduce the effort required from the end-user, making the system more efficient and user-friendly.
  • Negative Associations Matter: Exploiting negative feedback (cases that were not relevant) through pruning is as important as leveraging positive feedback for improving retrieval precision.
  • Decoupling Design and Deployment: The approach allows for a more practical development cycle, where an initial system can be deployed, and then improved iteratively based on real-world usage, rather than requiring constant redesign.

Key Results

The paper presents quantitative results demonstrating the effectiveness of the proposed techniques:

  • Recognizing Questions (short form): Achieved a precision of 66.42% and recall of 66.8% on queries with history, a significant improvement over the baseline case-base retrieval with interactive expansion (15.59% precision, 58.14% recall).
  • Automatic Question Expansion: Reduced user interaction significantly (from 18.7 features added interactively to 4.14 features automatically) while achieving 40.61% precision and 69.77% recall, outperforming interactive expansion.
  • Pruning: Showed improvements in precision across techniques, for example, increasing the precision of Automatic Expansion from 40.61% to 49.96% (pruned precision).
  • No Degradation on Unseen Data: Crucially, the history-based techniques did not degrade performance on questions with no relevant history.

Limitations and Future Work

  • User Effort for Feedback: While the paper suggests marking relevance adds little cost, it still requires user engagement. The authors acknowledge that if users are unwilling to provide feedback, improvement will be more gradual.
  • "Bad Guesses" in Long Form: The "long form" recognition (using interactively expanded queries) showed less impressive results, potentially due to variability introduced by user guesswork during expansion. This suggests the quality of user input for history is important.
  • Future Work: The authors suggest further exploration of how to automatically decipher the intended meaning of a query and how to integrate these techniques with other CBR concepts like adaptation and planning. They also note the potential for technical transfer to Information Retrieval (IR) but highlight key differences in data scale and the importance of adaptation in CBR.

Summary updated

BibTeX
@inproceedings{1996improving,
  title = {{Improving Case Retrieval by Remembering Questions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1996},
}
Figure