Classifying Texts Using Relevancy Signatures

Ellen Riloff, Wendy G. LehnertAAAI 1992

Introduction

In the early 1990s, the Message Understanding Conferences (MUC) highlighted a critical challenge in natural language processing: the difficulty of distinguishing relevant documents from irrelevant ones in complex, nuanced domains. In the "terrorism" domain, for instance, a report on a civilian killed by a bomb is relevant, while a soldier killed in a military skirmish is not. Traditional keyword-based information retrieval systems often fail because both scenarios share terms like "killed" and "bomb." Researchers at UMass Amherst proposed a solution: the Relevancy Signatures Algorithm, which employs a "text skimming" approach to achieve high-precision classification without exhaustive semantic analysis.

Methodology

The core of the approach is selective concept extraction. Instead of attempting to "understand" every word, the system focuses on specific linguistic patterns that reliably indicate relevance.

1. Relevancy Signatures

The basic algorithm uses the CIRCUS sentence analyzer to identify "concept nodes"—essentially case frames that represent events. A relevancy signature is defined as a pair: (trigger word, concept node). For example, the word "dead" might trigger a $found-dead-pass$ concept node. The system calculates a "reliability" score for each signature based on its historical frequency in relevant versus irrelevant training documents.

2. Augmented Relevancy Signatures

To handle more complex distinctions where the identity of the actor or victim is crucial, the authors introduced Augmented Relevancy Signatures. This version adds "slot filler" information, creating a triple: (concept node, slot name, semantic feature). For instance, a triple like (murder, victim, government-official) allows the system to distinguish between a relevant terrorist act and an irrelevant military engagement based on the victim's profile.

Experimental Setup

The algorithms were evaluated using the MUC-3 text corpus, comprising 1,500 news articles (wire stories, radio transcripts) concerning terrorism in Latin America. The researchers used 1,300 texts for training and two separate blocks of 100 texts for testing (DEV401-500 and DEV801-900). These test sets represented the "best" and "worst" performing blocks from preliminary runs to illustrate the system's performance range.

Results and Discussion

The experimental results demonstrated that the "skimming" approach can achieve near-perfect precision by focusing on highly reliable signatures.

Precision-Recall performance of the basic Relevancy Signatures algorithm on two test sets.

As shown in Figure 1, the basic Relevancy Signatures algorithm performed exceptionally well on the DEV401-500 set, maintaining 100% precision for recall levels up to 47%. However, performance dropped significantly on the more challenging DEV801-900 set, where precision peaked at only 67%.

Precision-Recall performance of the Augmented Relevancy Signatures algorithm showing improved stability and precision.

The introduction of Augmented Relevancy Signatures addressed this performance gap. As illustrated in Figure 2, the augmented version significantly boosted precision on the difficult DEV801-900 set, reaching 100% precision at 8% recall and 90% precision at 23% recall. This demonstrates that incorporating context—specifically who or what is involved in an event—is vital for stable, high-precision classification in complex domains.

Conclusion

The Relevancy Signatures Algorithm proves that machines can mimic human "skimming" behavior to identify relevant information with high accuracy. By automatically deriving rules from a tagged corpus, the system avoids the bottleneck of manual knowledge engineering. While the approach currently relies on a hand-crafted dictionary and requires a substantial training corpus for each new domain, it offers a powerful statistical shortcut for high-precision text classification in an era of increasing information density.

Summary updated

BibTeX
@inproceedings{riloff1992classifying,
  author = {Ellen Riloff and Wendy G. Lehnert},
  title = {{Classifying Texts Using Relevancy Signatures}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1992},
}
Figure