A Measure of Term Representativeness Based on the Number of Co-occurring Salient Words

Toru Hisamitsu, Yoshiki NiwaCOLING 2002

TL;DR

This paper introduces a novel method for measuring how representative a word is of a specific topic within a corpus. By focusing on "saliently biased" co-occurring words, their approach aims to be more robust and less susceptible to noise than existing methods, demonstrating superior performance in identifying topic-specific words.

What Problem Does This Paper Address?

In Natural Language Processing (NLP) and Information Retrieval (IR), understanding how well a word represents a particular topic (its "representativeness" or "topic specificity") is crucial for tasks like automatic dictionary creation and improving search interfaces. Existing methods, like TF-IDF, often struggle with words that appear frequently but are not truly indicative of a specific topic (e.g., common words like "do"). Furthermore, some methods are not well-suited for single words or have difficulty in setting appropriate thresholds for what constitutes a "representative" word. This paper addresses the need for a more accurate and robust measure of term representativeness that can effectively distinguish topic-specific words from general ones.

Methods

The core of this paper's contribution is a new measure called SAL(D(T), s). This measure quantifies term representativeness by counting the number of co-occurring words whose "saliency" exceeds a defined threshold 's'.

Here's a breakdown of the key methodological components:

  • Saliency Measure (HGS(w)): The paper defines the saliency of a word 'w' co-occurring with a term 'T' using a combinatorial probability measure, specifically a negative log of a hypergeometric distribution. This measure, denoted as HGS(w), quantifies how unlikely a word's occurrence is given the overall corpus distribution, thus indicating its "saliency" or "bias" within the context of term T.
  • SAL(D(T), s): This is the main proposed measure. It counts the number of distinct words 'w' within the set of documents containing term 'T' (D(T)) for which HGS(w) is greater than or equal to a threshold 's'. The intuition is that a truly representative term 'T' will co-occur with a higher number of saliently biased words.
  • Optimizing the Saliency Threshold (s): A critical aspect is automatically determining the optimal value for 's'. The authors propose a method that involves plotting SAL(D(T), s) against the size of the co-occurring document set (#Drand) for various 's' values. The optimal 's' is identified when the resulting curve is roughly parallel to the x-axis, indicating that the measure is no longer significantly dependent on the size of the co-occurring set (i.e., it's free from background noise). This allows for normalization-free representativeness.

Datasets

The experiments were conducted using a large archive of newspaper articles from the Nihon Keizai Shimbun (Nikkei). The dataset was referred to as "Do" and contained approximately 160,000 articles. For evaluating the effect of corpus size, two smaller subsets, "D1/2" (half the size of Do) and "D1/4" (one-quarter the size of Do), were also used.

Key Insights and Ideas

  • Focusing on "Salient" Co-occurrences is Key: The paper's central idea is that a representative term's co-occurring words should not just be frequent, but saliently biased compared to the general corpus. This focus helps filter out common, non-topic-specific words.
  • Automatic Threshold Optimization: The ability to automatically determine the optimal saliency threshold 's' is a significant advantage. This removes the need for manual tuning or ad-hoc threshold setting, making the measure more practical and reproducible.
  • Normalization-Free Measure: By finding the optimal 's', the authors achieve a measure (SAL(D(T), s_optimal)) that is inherently robust and doesn't require the complex baseline normalization used in prior work, leading to potentially higher precision.
  • Linguistic Intuition Alignment: The method's success in aligning with human linguistic intuition (e.g., "cipher" being more specific than "do") when traditional measures failed highlights its effectiveness.

Key Results

  • Superior Performance: The proposed SAL(D(T), s_optimal) measure consistently outperformed conventional measures like TF-IDF, NormDist(D(T)), and NormDIFFNUM(D(T)) in identifying topic-specific words.
  • DP-scores and ADP-scores: Figures 3 and 4 show that SAL(D(T), s_optimal) generally achieved higher DP-scores (measuring the accumulated number of positive class words) and ADP-scores (a cumulative measure of DP-scores), indicating better ranking of topic-specific words.
  • Robustness to Corpus Size: Experiments with smaller corpora (D1/2, D1/4) showed that SAL(D(T), s_optimal) remained superior to NormDist(D(T)), suggesting its effectiveness across different data scales. The optimal threshold 's' adjusted with corpus size, further validating the approach.

Limitations and Future Work

  • Computational Cost: While the paper mentions efficient calculation of the saliency measure using approximation techniques, the overall process of calculating SAL(D(T), s) for many terms and varying 's' might still be computationally intensive.
  • Definition of "Salient": The paper relies on a specific probabilistic definition of saliency. Exploring alternative definitions of saliency or bias could be a direction for future work.
  • Generalizability: While tested on newspaper archives, further evaluation on diverse text domains (e.g., scientific literature, social media) would be beneficial to confirm its broad applicability.
  • Multi-word Terms: The paper focuses on single terms. Extending the measure to handle multi-word terms effectively would be a valuable future direction.

Summary updated

BibTeX
@inproceedings{hisamitsu2002measure,
  author = {Toru Hisamitsu and Yoshiki Niwa},
  title = {{A Measure of Term Representativeness Based on the Number of Co-occurring Salient Words}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2002},
}
Figure