Introduction
Traditional Information Retrieval (IR) techniques, such as TF*IDF, are highly effective for long documents where statistical patterns are abundant. However, detecting semantic similarity between short text passages—sentences or paragraphs—presents a significant challenge due to data sparsity. When comparing two short segments, surface word overlap is often minimal or non-existent, even if the underlying meaning is identical.
This research addresses this "needle in a haystack" problem by moving beyond simple "aboutness." The authors define similarity strictly: two passages are similar only if they describe the exact same event or action. This distinction is vital for tasks like multi-document summarization, where identifying redundant information across sources is necessary to create concise summaries.
Methodology: Beyond Word Overlap
The researchers developed a machine-learning framework using the RIPPER rule-induction algorithm to combine diverse linguistic indicators into a robust similarity detector.
Linguistic Indicators
The system extracts two types of features:
- Primitive Features: These include basic elements like word co-occurrence (stems and parts of speech), Noun Phrase heads, WordNet synonyms, and Levin’s semantic verb classes.
- Composite Features: The core innovation involves combining primitive features with structural constraints. For instance, a composite feature might require a specific noun and verb to appear within a certain distance or in a specific order. This approximates grammatical relationships (e.g., subject-verb pairs) without the computational overhead of a full syntactic parse.
Normalization and Weighting
To ensure fairness across varying passage lengths, the authors applied length normalization. They also utilized a frequency-based weighting system, similar to Inverse Document Frequency (IDF), to prioritize rare, informative words over common ones.
Experimental Setup
The study utilized a subset of the 1997 TDT (Topic Detection and Tracking) pilot corpus, consisting of 30 Reuters articles. From these, 264 text units (paragraphs) were extracted, creating 10,345 unique pairs for comparison.
Ground truth was established by human reviewers who labeled pairs as "similar" or "not similar" based on the strict "same action" definition. Given that 97% of pairs were not similar, the dataset reflects the extreme imbalance typical of real-world similarity detection tasks.
Results and Analysis
The machine learning approach significantly outperformed traditional IR baselines. By leveraging linguistic features and structural constraints, the model maintained higher precision at comparable recall levels.
As shown in the performance summary, the ML approach achieved 60.5% precision at 36.6% recall, nearly doubling the precision of TF*IDF at a similar recall level. The Precision-Recall curve further illustrates the system's ability to maintain higher precision across various thresholds compared to standard methods.
Key Insights
- Noun Stability: Nouns proved to be more reliable indicators of similarity than verbs. While verbs often change meaning based on context, nouns typically name stable entities.
- Constraint-Based Precision: While individual features like a single verb match might be weak, combining them with distance or ordering constraints creates a high-precision signal.
- Task Distinction: The results confirm that "topical similarity" and "semantic similarity" are distinct computational tasks. Standard IR tools are optimized for the former, while the proposed linguistic features are necessary for the latter.
Conclusion and Future Work
This work demonstrates that combining deep linguistic features with machine learning can effectively overcome the data sparsity issues inherent in short-passage similarity detection. Future research directions include moving from pairwise comparison to clustering algorithms for grouping multiple similar units and exploring more complex features like full syntactic trees or semantic roles.
Summary updated