Information Weighted String and Semantic Similarity

This direction is merged into another direction: inverse frequency weighting. Its last version stays readable and citable here.

Version 1 of 1119 papers143 conceptsfrozen 2026-10-04

In natural language processing, determining if two pieces of text are "the same" is rarely a matter of simple identity. Language is redundant, noisy, and highly variable. If you compare two sentences by counting how many words they share, a match on the word "the" provides almost no evidence of shared meaning, while a match on a rare technical term like "neoplasm" is a smoking gun.

The core intuition behind this research direction is that similarity is not a flat sum of matches. Instead, the diagnostic value of a match is inversely proportional to its frequency. By weighting overlaps—whether they are characters, tokens, or semantic embeddings—by their "informativeness" or "rarity," researchers have created measures that align far more closely with human judgment and structural truth.

The Diagnostic Value of Rarity

The foundation of this idea dates back to string matching. Jörg Tiedemann argued in Automatic Construction of Weighted String Similarity Measures that we should learn the unique spelling "rules" of a language by assigning higher significance to rare characters. If two words in different languages share a rare character sequence, it is much stronger evidence of a shared origin than if they share common vowels.

This principle scales from characters to documents. In the realm of entity extraction, Zeyi Wen and Dong Deng 0001 demonstrated that applying Inverse Document Frequency (IDF) weights to tokens prevents common stop words from triggering expensive, irrelevant similarity computations, allowing systems to focus on the unique identifiers that actually define an entity (A Technical Report: Entity Extraction using Both Character-based and Token-based Similarity).

Weighting Semantic Overlap

As NLP moved from string matching to semantic similarity, the need for weighting remained. Rada Mihalcea and Courtney D. Corley pioneered methods to aggregate word-to-word similarity into sentence-level metrics. In Measuring the Semantic Similarity of Texts, they showed that weighting these matches by their information content ensures that rare, descriptive terms drive the final score, preventing common but low-meaning words from distorting the alignment.

This intuition is central to modern evaluation metrics like BERTScore. Tianyi Zhang showed that while contextual embeddings capture meaning, the final similarity score is significantly more robust when individual token contributions are adjusted by IDF weights (BERTScore: Evaluating Text Generation with BERT). This ensures that correctly generating a rare, content-bearing word is rewarded more than getting a functional particle right.

Information Theory as a Foundation

Why does this weighting work so consistently? Recent work suggests it is grounded in Information Theory. Hiroto Kurita and colleagues discovered that contrastive learning—the engine behind many modern sentence encoders—implicitly "reinvents" classic weighting schemes like TF-IDF. The model learns that rare words are more useful for distinguishing between similar and dissimilar pairs because they provide higher Information Gain (Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words).

Similarly, AlMotasem Bellah Al Ajlouni developed CEScore, which uses word entropy to measure meaning preservation. By prioritizing the retention of high-entropy (rare) words, the metric mirrors human priorities: we care much more if a simplification tool removes a specific noun than if it removes a generic conjunction (CEScore: Simple and Efficient Confidence Estimation Model for Evaluating Split and Rephrase).

Domain-Specific Informativeness

The definition of "informative" changes depending on the domain. In legal retrieval, the most informative units are often not words, but citations. Haitao Li and Qingyao Ai developed the LeCaRDv2 dataset and the Caseformer model, which treat citations of rare law articles as the primary signal for similarity (LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset). Just as rare words identify a topic, rare statutes identify a specific legal logic.

In computer vision, Lianli Gao applied this to scene graphs. Their BA-SGG framework weights the recall of predicted relationships by their information content (negative log-likelihood). This forces models to move beyond "lazy" generic predicates like "on" and rewards them for identifying rare, descriptive relationships like "parked on" (From General to Specific: Informative Scene Graph Generation via Balance Adjustment).

Efficiency and Scaling

Information weighting isn't just about accuracy; it's also a tool for efficiency. By focusing on high-information tokens, researchers can prune search spaces and speed up computations:

By treating every token not as a unit of text, but as a unit of information, this research direction provides a robust framework for navigating the inherent messiness of human language.

Papers in this version

Showing 60 of 119 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_information_weighted_string_and_semantic_74b90eb2,
  title = {Information Weighted String and Semantic Similarity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-weighted-string-and-semantic-similarity/rdr_74b90eb2e7d74a8fbab4a49bb988fa02},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_information_weighted_string_and_semantic_74b90eb2_v1,
  title = {Information Weighted String and Semantic Similarity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-weighted-string-and-semantic-similarity/rdr_74b90eb2e7d74a8fbab4a49bb988fa02/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}