Learning Term-weighting Functions for Similarity Measures

Wen-tau YihEMNLP 2009 · OpenReview

Abstract

Measuring the similarity between two texts is a fundamental problem in many NLP and IR applications. Among the existing approaches, the cosine measure of the term vectors representing the original texts has been widely used, where the score of each term is often determined by a TFIDF formula. Despite its simplicity, the quality of such cosine similarity measure is usually domain dependent and decided by the choice of the term-weighting function. In this paper, we propose a novel framework that learns the term-weighting function. Given the labeled pairs of texts as training data, the learning procedure tunes the model parameters by minimizing the specified loss function of the similarity score. Compared to traditional TFIDF term-weighting schemes, our approach shows a significant improvement on tasks such as judging the quality of query suggestions and filtering irrelevant ads for online advertising.

Introduction

In Natural Language Processing (NLP) and Information Retrieval (IR), measuring the similarity between two texts is a foundational task. The industry standard has long been the vector space model: representing texts as vectors of terms, weighting those terms (traditionally with TF-IDF), and calculating the cosine of the angle between the vectors.

However, this approach has two major flaws. First, it is domain-insensitive; TF-IDF is a "one-size-fits-all" formula that ignores structural information, such as whether a word appears in a web page's title or footer. Second, it is rigid; incorporating new information, such as HTML tags or anchor text, into a traditional TF-IDF formula is mathematically cumbersome. This paper introduces TWEAK (Term-WEighting learning frAmework), a data-driven approach that learns the optimal way to weight terms based on specific task requirements.

The TWEAK Framework

Instead of a fixed formula, TWEAK defines the weight of a term $t_i$ in document $D_p$ as a linear combination of features: $$s_p^i = \sum_j w_j \phi_j(t_i, D_p)$$ where $\phi_j$ are features of the term (e.g., frequency, position, capitalization) and $w_j$ are the parameters the model learns. By learning weights for features rather than specific words, the model generalizes to unseen vocabulary and avoids data sparsity issues.

The TWEAK framework calculates similarity by learning feature-based term weights.

The authors explore different learning objectives for optimization:

  1. Metric Learning (SSE & Log-loss): Predicting a specific similarity score for a pair of documents.
  2. Preference Ordering: A "learning to rank" approach where the model ensures that a "more similar" pair receives a higher score than a "less similar" pair.

Experimental Setup

The framework was evaluated on two real-world tasks:

  • Query Suggestion: 4,852 query/suggestion pairs from a commercial search engine, labeled on a 4-point scale (Excellent, Good, Fair, Bad).
  • Ad Relevance: 13,341 pairs of search queries and ad landing pages. This dataset included rich HTML features like URL, Title, Meta tags, and Anchor text.

Results and Discussion

The experimental results demonstrate that learning term-weighting functions significantly outperforms traditional methods. On the query suggestion task, the Preference-based TWEAK model achieved an AUC of 0.782, compared to 0.732 for the Web-kernel baseline.

Comparison of TWEAK variants against the Web-kernel baseline on the query suggestion task.

Key insights from the evaluation include:

  • Preference Learning is Superior: Training the model to respect the relative order of similarity (Preference Ordering) proved more effective than regressing to absolute scores. This aligns with the nature of human similarity judgments, which are often relative.
  • HTML Structural Features: For web-based tasks, the model successfully learned that terms in the Title, URL, and Meta tags are significantly more indicative of relevance than general body text. Using these features, the model reached an AUC of 0.855 in the ad filtering task.
  • Generalization: Even when restricted to the same basic information as TF-IDF (term and document frequency), the learned TWEAK function outperformed the standard formula, suggesting that traditional log-based curves are not necessarily optimal for all domains.

Conclusion

TWEAK provides a flexible, principled framework for learning task-specific similarity measures. While the current study focuses on linear combinations and cosine similarity, future work could explore non-linear functions and other vector operations. The success in query suggestion and ad filtering suggests broad applicability across other similarity-based NLP tasks like paraphrase recognition or duplicate detection.

Summary updated

Abstract

Measuring the similarity between two texts is a fundamental problem in many NLP and IR applications. Among the existing approaches, the cosine measure of the term vectors representing the original texts has been widely used, where the score of each term is often determined by a TFIDF formula. Despite its simplicity, the quality of such cosine similarity measure is usually domain dependent and decided by the choice of the term-weighting function. In this paper, we propose a novel framework that learns the term-weighting function. Given the labeled pairs of texts as training data, the learning procedure tunes the model parameters by minimizing the specified loss function of the similarity score. Compared to traditional TFIDF term-weighting schemes, our approach shows a significant improvement on tasks such as judging the quality of query suggestions and filtering irrelevant ads for online advertising.
BibTeX
@inproceedings{yih2009learning,
  author = {Wen-tau Yih},
  title = {{Learning Term-weighting Functions for Similarity Measures}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
  url = {https://openreview.net/forum?id=SkZBgffdWr},
}
Figure