Detecting Text Similarity over Short Passages: Exploring Linguistic Feature Combinations via Machine Learning

Vasileios Hatzivassiloglou, Judith L. Klavans, Eleazar EskinEMNLP 1999

Introduction

Traditional Information Retrieval (IR) techniques, such as TF*IDF, are highly effective for long documents where statistical patterns are abundant. However, detecting semantic similarity between short text passages—sentences or paragraphs—presents a significant challenge due to data sparsity. When comparing two short segments, surface word overlap is often minimal or non-existent, even if the underlying meaning is identical.

This research addresses this "needle in a haystack" problem by moving beyond simple "aboutness." The authors define similarity strictly: two passages are similar only if they describe the exact same event or action. This distinction is vital for tasks like multi-document summarization, where identifying redundant information across sources is necessary to create concise summaries.

Methodology: Beyond Word Overlap

The researchers developed a machine-learning framework using the RIPPER rule-induction algorithm to combine diverse linguistic indicators into a robust similarity detector.

Linguistic Indicators

The system extracts two types of features:

  1. Primitive Features: These include basic elements like word co-occurrence (stems and parts of speech), Noun Phrase heads, WordNet synonyms, and Levin’s semantic verb classes.
  2. Composite Features: The core innovation involves combining primitive features with structural constraints. For instance, a composite feature might require a specific noun and verb to appear within a certain distance or in a specific order. This approximates grammatical relationships (e.g., subject-verb pairs) without the computational overhead of a full syntactic parse.

Normalization and Weighting

To ensure fairness across varying passage lengths, the authors applied length normalization. They also utilized a frequency-based weighting system, similar to Inverse Document Frequency (IDF), to prioritize rare, informative words over common ones.

Experimental Setup

The study utilized a subset of the 1997 TDT (Topic Detection and Tracking) pilot corpus, consisting of 30 Reuters articles. From these, 264 text units (paragraphs) were extracted, creating 10,345 unique pairs for comparison.

Ground truth was established by human reviewers who labeled pairs as "similar" or "not similar" based on the strict "same action" definition. Given that 97% of pairs were not similar, the dataset reflects the extreme imbalance typical of real-world similarity detection tasks.

Results and Analysis

The machine learning approach significantly outperformed traditional IR baselines. By leveraging linguistic features and structural constraints, the model maintained higher precision at comparable recall levels.

Comparison of the proposed machine learning approach against traditional IR baselines (TF*IDF and SMART) and a default baseline.

As shown in the performance summary, the ML approach achieved 60.5% precision at 36.6% recall, nearly doubling the precision of TF*IDF at a similar recall level. The Precision-Recall curve further illustrates the system's ability to maintain higher precision across various thresholds compared to standard methods.

Precision-Recall curve comparing the performance of the linguistic machine learning system (solid line) against the TF*IDF baseline (dotted line).

Key Insights

  • Noun Stability: Nouns proved to be more reliable indicators of similarity than verbs. While verbs often change meaning based on context, nouns typically name stable entities.
  • Constraint-Based Precision: While individual features like a single verb match might be weak, combining them with distance or ordering constraints creates a high-precision signal.
  • Task Distinction: The results confirm that "topical similarity" and "semantic similarity" are distinct computational tasks. Standard IR tools are optimized for the former, while the proposed linguistic features are necessary for the latter.

Conclusion and Future Work

This work demonstrates that combining deep linguistic features with machine learning can effectively overcome the data sparsity issues inherent in short-passage similarity detection. Future research directions include moving from pairwise comparison to clustering algorithms for grouping multiple similar units and exploring more complex features like full syntactic trees or semantic roles.

Summary updated

BibTeX
@inproceedings{hatzivassiloglou1999detecting,
  author = {Vasileios Hatzivassiloglou and Judith L. Klavans and Eleazar Eskin},
  title = {{Detecting Text Similarity over Short Passages: Exploring Linguistic Feature Combinations via Machine Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {1999},
}
Figure