In natural language processing, determining if two pieces of text are "the same" is rarely a matter of simple identity. Language is redundant, noisy, and highly variable. If you compare two sentences by counting how many words they share, a match on the word "the" provides almost no evidence of shared meaning, while a match on a rare technical term like "neoplasm" is a smoking gun.
The core intuition behind this research direction is that similarity is not a flat sum of matches. Instead, the diagnostic value of a match is inversely proportional to its frequency. By weighting overlaps—whether they are characters, tokens, or semantic embeddings—by their "informativeness" or "rarity," researchers have created measures that align far more closely with human judgment and structural truth.
The Diagnostic Value of Rarity
The foundation of this idea dates back to string matching. Jörg Tiedemann argued in Automatic Construction of Weighted String Similarity Measures that we should learn the unique spelling "rules" of a language by assigning higher significance to rare characters. If two words in different languages share a rare character sequence, it is much stronger evidence of a shared origin than if they share common vowels.
This principle scales from characters to documents. In the realm of entity extraction, Zeyi Wen and Dong Deng 0001 demonstrated that applying Inverse Document Frequency (IDF) weights to tokens prevents common stop words from triggering expensive, irrelevant similarity computations, allowing systems to focus on the unique identifiers that actually define an entity (A Technical Report: Entity Extraction using Both Character-based and Token-based Similarity).
Weighting Semantic Overlap
As NLP moved from string matching to semantic similarity, the need for weighting remained. Rada Mihalcea and Courtney D. Corley pioneered methods to aggregate word-to-word similarity into sentence-level metrics. In Measuring the Semantic Similarity of Texts, they showed that weighting these matches by their information content ensures that rare, descriptive terms drive the final score, preventing common but low-meaning words from distorting the alignment.
This intuition is central to modern evaluation metrics like BERTScore. Tianyi Zhang showed that while contextual embeddings capture meaning, the final similarity score is significantly more robust when individual token contributions are adjusted by IDF weights (BERTScore: Evaluating Text Generation with BERT). This ensures that correctly generating a rare, content-bearing word is rewarded more than getting a functional particle right.
Information Theory as a Foundation
Why does this weighting work so consistently? Recent work suggests it is grounded in Information Theory. Hiroto Kurita and colleagues discovered that contrastive learning—the engine behind many modern sentence encoders—implicitly "reinvents" classic weighting schemes like TF-IDF. The model learns that rare words are more useful for distinguishing between similar and dissimilar pairs because they provide higher Information Gain (Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words).
Similarly, AlMotasem Bellah Al Ajlouni developed CEScore, which uses word entropy to measure meaning preservation. By prioritizing the retention of high-entropy (rare) words, the metric mirrors human priorities: we care much more if a simplification tool removes a specific noun than if it removes a generic conjunction (CEScore: Simple and Efficient Confidence Estimation Model for Evaluating Split and Rephrase).
Domain-Specific Informativeness
The definition of "informative" changes depending on the domain. In legal retrieval, the most informative units are often not words, but citations. Haitao Li and Qingyao Ai developed the LeCaRDv2 dataset and the Caseformer model, which treat citations of rare law articles as the primary signal for similarity (LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset). Just as rare words identify a topic, rare statutes identify a specific legal logic.
In computer vision, Lianli Gao applied this to scene graphs. Their BA-SGG framework weights the recall of predicted relationships by their information content (negative log-likelihood). This forces models to move beyond "lazy" generic predicates like "on" and rewards them for identifying rare, descriptive relationships like "parked on" (From General to Specific: Informative Scene Graph Generation via Balance Adjustment).
Efficiency and Scaling
Information weighting isn't just about accuracy; it's also a tool for efficiency. By focusing on high-information tokens, researchers can prune search spaces and speed up computations:
- Near-Duplicate Detection: Yuheng Zhang and Miao Qiao showed that incorporating TF-IDF weights into Jaccard similarity significantly reduces false positives and allows for faster indexing (Near-Duplicate Text Alignment under Weighted Jaccard Similarity).
- Data Pruning: Minsang Kim demonstrated that LLMs can be trained on 50% less data by ranking samples by their "informativeness"—a combination of model surprise and word rarity—actually improving generalization by removing low-information noise (Measuring Sample Importance in Data Pruning for Training LLMs from a Data Compression Perspective).
By treating every token not as a unit of text, but as a unit of information, this research direction provides a robust framework for navigating the inherent messiness of human language.