TL;DR
This paper introduces a "baseline method" for measuring how representative or domain-specific a term is. By normalizing a term's characteristic value against a baseline derived from random document sets, their method effectively identifies informative terms and overcomes limitations of traditional measures like TF-IDF, particularly in handling frequent but uninformative words.
What Problem Does This Paper Address?
In natural language processing (NLP) and information retrieval (IR), identifying "representative" terms is crucial. These are words that are particularly informative or specific to a particular domain or document set. For instance, in an IR system that provides an overview of retrieved documents, representative words help users quickly grasp the content. However, existing methods struggle with several issues:
- Filtering Uninformative Words: Classical measures like TF-IDF are too sensitive to word frequency and fail to automatically filter out common, uninformative words (e.g., "year," "month"). Manually creating stop-word lists is a laborious and often incomplete solution.
- Distinguishing Representativeness: Current measures often don't adequately differentiate between terms with subtle differences in representativeness. For example, distinguishing between a highly specific term like "cipher" and a more general one like "read" can be difficult.
- Domain Specificity: Some methods require pre-categorized documents, limiting their applicability.
- Single Word Terms: Many NLP-focused measures rely on multi-word sequences (n-grams) and struggle with single words.
- Ad Hoc Thresholds: Determining a reliable threshold for what constitutes a "representative" term is often arbitrary.
This paper aims to address these limitations by proposing a more robust and systematic method for measuring term representativeness.
Methods
The core contribution is the "baseline method" for measuring term representativeness. The fundamental idea is captured by Firth's quote: "You shall know a word by the company it keeps."
Here's how it works:
- Define a Characteristic Measure (M): A measure is chosen to characterize a document set. The paper explores two:
- DIFFNUM: The number of distinct words in a document set.
- LLR (Log-Likelihood Ratio): The distance between the word distribution of a specific document set (containing a term T, denoted D(T)) and the word distribution of the entire corpus (D₀). This is the measure that ultimately shows superior performance.
- Calculate M(D(T)): The chosen measure M is calculated for the document set D(T) that contains the term T.
- Estimate a Baseline Function (B): A "baseline function" is constructed. This function estimates the expected value of M for a randomly chosen document set of the same size as D(T). This is done by:
- Sampling random document sets of varying sizes from a corpus.
- Calculating M for each random set.
- Approximating the relationship between document set size and M using logarithmic linear approximation, creating a "baseline curve."
- Normalize: The representativeness of term T, denoted Rep(T, M), is calculated by normalizing M(D(T)) using the baseline function B(#D(T)), where #D(T) is the size of the document set D(T). The formula for LLR is:
Rep(T, LLR) = 100 × (log(LLR(D(T))) / log(BLLR(#D(T))) – 1)
This normalization accounts for the natural tendency of measures like LLR to increase with document set size, allowing for fair comparison across terms with different document frequencies.
Datasets
The experiments primarily used the Nikkei Shinbun (Japanese economic newspaper) corpus from 1996. Various subsets of this corpus were used to construct baseline functions and evaluate measures. For evaluation, a set of 20,000 words was randomly selected from 86,000 words with document frequencies greater than 2. These words were manually classified into three groups:
- Class a (acceptable): Useful words for a navigation window.
- Class d (delete): Words not useful.
- Class u (uncertain): Words whose usefulness was neutral or difficult to judge.
Additionally, the NACSIS corpus was used in experiments on baseline function portability.
Key Insights and Ideas
- Normalization is Key: The raw LLR values are not directly comparable because they increase with document set size. The baseline method's normalization step is crucial for making meaningful comparisons.
- LLR Outperforms DIFFNUM: The normalized log-likelihood ratio (Rep(•, LLR)) proved to be a superior measure of representativeness compared to the number of distinct words (Rep(•, DIFFNUM)). This suggests that capturing the distributional difference of words is more informative than just counting unique words.
- Baseline Portability: A significant finding is that a baseline function constructed from one corpus can be effectively used to normalize measures in a different corpus, even if the corpora have different sizes or domains. This makes the method highly practical.
- Systematic Thresholding: The baseline method allows for a systematic definition of a threshold for representativeness, moving away from ad hoc choices.
- Handling Frequent Terms: The method is effective at identifying and down-weighting frequent but uninformative terms, a major weakness of traditional methods.
Key Results
- Superiority of Rep(•, LLR): In experiments evaluating the ability to identify "acceptable" words (class a), Rep(•, LLR) significantly outperformed TF-IDF, frequency, and Rep(•, DIFFNUM). For example, using Rep(•, LLR), the top 1,511 words achieved 85% recall and 50% precision for class a words.
- Effective Filtering of Uninformative Words: Rep(•, LLR) was particularly effective at identifying "delete" words (class d), significantly outperforming other measures.
- Baseline Portability Confirmed: Experiments showed that using baseline functions from different corpora resulted in very similar rankings of terms, with rank correlations close to 100% in most cases.
- Low Correlation Between Measures: The rank correlation between Rep(•, LLR) and Rep(•, DIFFNUM) was very low, suggesting that combining these measures could offer even stronger discriminative power.
Limitations and Future Work
- Limitations:
- The paper notes that while Rep(•, DIFFNUM) outperformed Rep(•, LLR) in picking out frequent non-representative terms in some specific scenarios, LLR generally showed better overall performance.
- The paper acknowledges that the baseline function construction involves approximations and part-wise linear fitting.
- Future Work:
- Theoretical Estimation of Baseline Functions: The authors suggest theoretically estimating baseline functions using fundamental corpus parameters like total word count and distinct word count.
- Application to IR Tasks: Applying Rep(•, LLR) and Rep(•, DIFFNUM) to tasks like automatic stop-word list construction and term weighting in document similarity calculation.
- Investigating Combined Measures: Further exploring the potential of combining Rep(•, LLR) and Rep(•, DIFFNUM) for enhanced term classification.
- Further Study of Baseline Functions: Deeper theoretical investigation into the nature of baseline functions.
Summary updated