TL;DR
This paper introduces "Inverse Category Frequency" (ICF) as a novel term weighting strategy for text categorization, arguing that a term's importance is better judged by how few categories it appears in, rather than how few documents it appears in (as in traditional TF-IDF). They propose two new methods, tf.icf for multi-class and icf-based for binary classification, which consistently outperform existing methods across various datasets and classifiers.
What Problem Does This Paper Address?
Text categorization (TC), the task of assigning documents to predefined categories, heavily relies on how text is represented. A crucial part of this representation is "term weighting," which assigns importance scores to words. The standard approach, TF-IDF (Term Frequency-Inverse Document Frequency), borrows from information retrieval. However, the authors argue that TF-IDF's "Inverse Document Frequency" component is less intuitive for TC. In TC, we have access to category labels during training, which TF-IDF doesn't fully leverage. The core problem is that TF-IDF might over- or under-weight terms because it only considers their global document frequency, ignoring their distribution across different categories. This can lead to suboptimal classification performance, especially when dealing with imbalanced datasets or when a term's discriminative power is tied to its presence in a limited set of categories.
Methods
The paper introduces two novel term weighting schemes:
tf.icf(Term Frequency-Inverse Category Frequency): This method is designed for multi-class classification. It replaces the Inverse Document Frequency (IDF) component of TF-IDF with Inverse Category Frequency (ICF). The ICF of a term is calculated based on how many categories it appears in. The intuition is that a term appearing in fewer categories is more discriminative for those specific categories. This approach leverages the category information available in TC.icf-based(ICF-based Supervised Term Weighting Scheme): This method is for binary classification and is a supervised approach. It combines ICF with Relevance Frequency (RF), which considers the distribution of a term in positive and negative categories. This scheme aims to provide a more nuanced weighting by incorporating both category-level rarity (ICF) and class-specific relevance (RF).
The authors also discuss existing unsupervised (e.g., TF-IDF) and supervised (e.g., TF-RF, prob-based) term weighting schemes to provide context and comparison.
Datasets
The experiments were conducted on three widely used text corpora:
- Reuters-21578: A skewed, single-label multi-class dataset. The authors used a subset called Reuters-52.
- 20 Newsgroup: A balanced dataset with twenty categories.
- La12: A large dataset derived from TREC.
The paper does not introduce new benchmarks but uses these established datasets to evaluate the proposed methods against existing ones.
Key Insights and Ideas
- ICF is more intuitive for TC than IDF: The core insight is that a term's discriminative power in text categorization is better captured by its rarity across categories (ICF) rather than its rarity across documents (IDF). A term appearing in only one or two categories is likely a stronger indicator of those categories than a term appearing in many documents but spread across numerous categories.
- Supervised methods can leverage category information better: By using category labels during training, supervised methods can achieve superior performance. The
icf-basedmethod specifically combines ICF with RF to exploit this information effectively. - The "distribution of a term among categories disappears" in some supervised methods: The authors highlight a limitation in some existing supervised methods (like TF-RF and prob-based) where the multi-class nature of the problem is simplified into binary classifications, losing information about a term's distribution across all categories. Their ICF-based approach aims to address this.
- SVM performance is less sensitive to term weighting than other classifiers: The experiments showed that while term weighting is important, SVM classifiers, particularly with RBF kernels, are less affected by the choice of weighting scheme compared to kNN or centroid-based classifiers.
Key Results
tf.icfconsistently outperforms traditional UTWS: Across all three datasets and various classifiers (SVM, kNN, Centroid),tf.icfachieved better or comparable macro-F1 and micro-F1 scores compared to TF, IDF, and TF-IDF. For instance, on Reuters-52,tf.icfimproved macro-F1 by up to 3.3% over TF-IDF with a centroid classifier.icf-basedoutperforms existing STWS: For binary classification tasks, the proposedicf-basedmethod consistently outperformed eight other supervised term weighting schemes, including TF-RF and prob-based, across the tested corpora. For example, on Reuters-52,icf-basedachieved a macro-F1 of 54.7%, slightly better than TF-RF.- Significant improvements with kNN and Centroid classifiers: The benefits of the proposed ICF-based methods were particularly pronounced when used with kNN and centroid-based classifiers, which are known to be more sensitive to term weighting schemes.
Limitations and Future Work
- Limitations:
- The paper notes that while
icf-basedis generally superior, traditional TF-IDF can sometimes perform better, especially on balanced datasets like 20 Newsgroup with kNN or centroid classifiers.
- The paper notes that while
- The
icf-basedmethod's performance can be affected by imbalanced datasets, particularly when using SVM with default parameters, as SVM is a global optimization scheme. - The authors mention that the "a/b" factor in the prob-based scheme, representing the ratio of documents with and without a term in a category, lacks intuitive explanation and can negatively impact performance.
- Future Work:
- The authors suggest conducting further experiments to validate the generalization of inverse category frequency for text categorization tasks.
Summary updated