Semi-supervised Learning with Weakly-Related Unlabeled Data: Towards Better Text Categorization

Liu Yang, Rong Jin, Rahul SukthankarNeurIPS 2008 · OpenReview

Overview

Standard semi-supervised learning (SSL) often fails when unlabeled data doesn't perfectly match the target topic. This paper introduces SSLW (Semi-supervised Learning with Weakly-Related Unlabeled Data), a method that extracts universal word-usage patterns from "weakly-related" external corpora to build better text classifiers, even when you only have a handful of labeled examples.

The Problem: The Flaw in the Cluster Assumption

Semi-supervised learning usually relies on the "cluster assumption": the idea that the decision boundary between classes should fall in "empty" spaces (low-density regions) of the unlabeled data. This works exceptionally well if your unlabeled data comes from the exact same distribution as your labeled data.

However, in the real world, finding perfectly matched unlabeled data is difficult. If you are trying to classify specific medical papers but use general news articles as your unlabeled resource, the cluster assumption breaks down. Forcing a decision boundary to avoid the high-density areas of unrelated news might actually lead the classifier astray. The authors tackle a critical question: How can we benefit from massive amounts of unlabeled data that are only tangentially related to our specific task?

Methods: Leveraging Feature Coherence

The authors propose SSLW to address this gap. Instead of looking for gaps in data density, SSLW looks for feature coherence.

The core intuition is that while topics change across different corpora, the way words relate to each other (word correlation) remains relatively consistent across a language. For example, "wheat" and "farm" will likely co-occur in both a specialized agricultural database and a general news corpus.

Key technical steps include: 1. Word Correlation Matrix ($R$): Instead of using a standard fixed kernel for a Support Vector Machine (SVM), SSLW learns an optimal word correlation matrix. 2. Regularization via External Data: It uses the co-occurrence patterns from the weakly-related unlabeled data to constrain this matrix. 3. Unified Framework: Unlike "self-taught learning" which performs feature extraction and classification in two separate steps, SSLW solves for the correlation matrix and the classification margin simultaneously. 4. Optimization: The authors formulate this as a convex optimization problem. Specifically, they transform a complex Semi-Definite Programming (SDP) problem into a Second Order Cone Programming (SOCP) problem, which is much more computationally efficient and scalable.

Experimental Setup

The researchers evaluated SSLW on two classic text categorization benchmarks:

  • Reuters-21578: The 10 largest categories (e.g., earn, acq, grain).
  • WebKB: Six categories of university webpages (e.g., student, faculty, project).

To simulate real-world conditions with "weakly-related" sources, they used 1,000 documents from the TREC AP88 dataset (Associated Press news from 1988) as the unlabeled resource for both tasks. This data has very little topical overlap with the target categories.

Key Insights

  • The Cluster Assumption is a Double-Edged Sword: The paper demonstrates that traditional SSL methods (like TSVM) can actually perform worse than a basic supervised SVM if the unlabeled data is poorly matched.
  • Word Patterns > Topic Distributions: Even if an unlabeled document isn't about your target topic, the "co-occurrence" of words within it provides a valuable prior for how the language works, helping the model generalize from very few labeled examples.
  • Stability in Scarcity: SSLW significantly reduces the variance of the results. When you only have a few training documents (e.g., 4 positive, 4 negative), standard models are highly sensitive to which examples you pick. SSLW uses the external data to "anchor" the model, leading to much more reliable performance.

Key Results

The authors measured performance using the Area Under the ROC Curve (AUR). SSLW was compared against six state-of-the-art baselines, including standard SVM, TSVM, and Manifold Regularization.

WebKB Categorization Results (AUR)
Reuters Categorization Results (AUR)

As seen in the tables above, SSLW demonstrates:

  • Consistent Dominance: SSLW outperformed the baselines across almost all categories in both the WebKB and Reuters datasets.
  • Significant Gains: In categories like "wheat" and "ship" within the Reuters dataset, SSLW improved AUR by more than 10% compared to the next best method.
  • Reliability: SSLW maintained a standard deviation of less than 2.5% in most cases, whereas baseline methods often saw fluctuations above 5%, proving it is more robust when training data is limited.

Limitations and Future Work

  • Hyperparameter Sensitivity: The method relies on selecting the top $p$ eigenvectors (the authors used 200) from the document-word matrix. The paper suggests further investigation into how this choice impacts performance.
  • Domain Specificity: While this works well for text (where word correlation is a strong signal), it's unclear how well this specific formulation translates to other domains like computer vision or audio processing without similar "universal" feature relationships.
  • Scaling: While SOCP is more efficient than SDP, scaling to truly massive datasets and exploring the "diminishing returns" as more labeled data becomes available remain important next steps.

Summary updated

Abstract

The cluster assumption is exploited by most semi-supervised learning (SSL) methods. However, if the unlabeled data is merely weakly related to the target classes, it becomes questionable whether driving the decision boundary to the low density regions of the unlabeled data will help the classification. In such case, the cluster assumption may not be valid; and consequently how to leverage this type of unlabeled data to enhance the classification accuracy becomes a challenge. We introduce Semi-supervised Learning with Weakly-Related Unlabeled Data" (SSLW), an inductive method that builds upon the maximum-margin approach, towards a better usage of weakly-related unlabeled information. Although the SSLW could improve a wide range of classification tasks, in this paper, we focus on text categorization with a small training pool. The key assumption behind this work is that, even with different topics, the word usage patterns across different corpora tends to be consistent. To this end, SSLW estimates the optimal word-correlation matrix that is consistent with both the co-occurrence information derived from the weakly-related unlabeled documents and the labeled documents. For empirical evaluation, we present a direct comparison with a number of state-of-the-art methods for inductive semi-supervised learning and text categorization; and we show that SSLW results in a significant improvement in categorization accuracy, equipped with a small training set and an unlabeled resource that is weakly related to the test beds."
BibTeX
@inproceedings{yang2008semi,
  author = {Liu Yang and Rong Jin and Rahul Sukthankar},
  title = {{Semi-supervised Learning with Weakly-Related Unlabeled Data: Towards Better Text Categorization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2008},
  url = {https://openreview.net/forum?id=SyNBd_ZdZH},
}
Figure