A Rule-Based Approach to Information Retrieval: Some Results and Comments

Richard M. Tong, Daniel G. Shapiro, Brian P. McCune, Jeffrey S. DeanAAAI 1983

Abstract

This paper presents RUBRIC (Rule-Based Retrieval of Information by Computer), a prototype system that conceptualizes information retrieval as an expert system task. Moving beyond the rigid constraints of Boolean keyword searches, RUBRIC employs hierarchical production rules and fuzzy uncertainty values to capture the nuances of human relevance judgments. The system allows for partial matches and incorporates auxiliary evidence to refine retrieval accuracy. Experimental results on a news dataset demonstrate that RUBRIC provides superior ranking and flexibility compared to traditional Boolean logic.

Introduction

In the early 1980s, information retrieval (IR) was primarily divided between rigid Boolean keyword systems and complex statistical models. Boolean systems, while precise, often failed to capture context and provided only binary "yes/no" results. Statistical systems were powerful but often opaque to the end-user. RUBRIC was developed to bridge this gap, providing a way for "naive" users to search using conceptual vocabularies that handle the inherent imprecision of language.

The RUBRIC System

The core innovation of RUBRIC is its representation of queries as a hierarchy of concepts and sub-concepts. Instead of a flat list of keywords, a user defines a top-level topic which is decomposed into constituent parts via production rules.

Rule-Based Hierarchy and Fuzzy Logic

Each rule in the hierarchy is assigned a heuristic weight (ranging from 0 to 1), representing the degree of confidence that the evidence supports the conclusion. This allows the system to calculate a graded measure of relevance rather than a simple binary match.

Hierarchical rule structure for the 'Terrorism' query in RUBRIC, illustrating how concepts like 'Violent-Event' and 'Reason' contribute to the top-level topic.

A unique feature of RUBRIC is its ability to handle auxiliary evidence. Rules can be modified based on context; for instance, the presence of specific secondary terms can increase or decrease the confidence of a primary rule. At the lowest level, the system utilizes a text reference language to identify patterns within specific proximity constraints (e.g., within the same sentence).

Experimental Evaluation

The authors evaluated RUBRIC using a controlled dataset consisting of 30 stories from the Reuters News Service. These stories were manually categorized into a "subjective ground truth" regarding the topic of "Acts of Terrorism" (9 relevant, 4 marginally relevant, and 17 irrelevant).

Results and Uncertainty Calculi

The system's performance was compared against standard Boolean logic. While a strict Boolean "AND" query missed several relevant stories, a broad "OR" query introduced significant noise. RUBRIC successfully prioritized relevant content through its nuanced ranking system.

Distribution of story ratings for relevant and non-relevant documents, showing the application of precision and recall thresholds.

The researchers also explored various mathematical "calculi" for propagating weights through the rule tree. After testing 20 different methods, they found that while "min-max" and product-based logics performed well, the optimal calculus often depended on the user's specific rule-writing style—specifically whether they intended evidence to be cumulative or alternative.

Discussion and Key Insights

Several critical insights emerged from the RUBRIC prototype:

  • Disconfirming Evidence: The system can effectively use negative context (e.g., identifying "bomb" in a sports context) to reduce false positives.
  • The Vanishing Relevance Problem: In deep hierarchies, relevance scores tend to diminish. The authors concluded that relative ranking is more valuable to users than absolute numerical scores.
  • Symbolic Uncertainty: Future iterations should move away from manual numerical weighting toward symbolic labels like "strong evidence," making the system more intuitive for non-experts.

Conclusion

RUBRIC demonstrates that a rule-based, expert system approach can significantly enhance the flexibility and precision of information retrieval. By allowing users to define complex, hierarchical concepts and accounting for uncertainty, the system provides a more human-centric way to navigate large document repositories.

Summary updated

BibTeX
@inproceedings{tong1983rule,
  author = {Richard M. Tong and Daniel G. Shapiro and Brian P. McCune and Jeffrey S. Dean},
  title = {{A Rule-Based Approach to Information Retrieval: Some Results and Comments}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1983},
}
Figure