X-Class: Text Classification with Extremely Weak Supervision

Zihan Wang, Dheeraj Mekala, Jingbo Shang

Abstract

Zihan Wang, Dheeraj Mekala, Jingbo Shang. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2021.

Introduction

Traditional text classification models rely heavily on large-scale labeled datasets, which are expensive and time-consuming to produce. While "weak supervision" methods—using a few labeled examples or expert-curated "seed words"—have mitigated this burden, they still require domain expertise to select effective seeds. X-Class introduces the concept of Extremely Weak Supervision, a framework that performs text classification using only the surface text of class names (e.g., "Sports," "Politics").

The core challenge in this setting is that standard document representations, such as simple averages of BERT embeddings, are often biased toward general topics. A representation that works for topic classification might fail for sentiment or location classification because it is not "adaptive" to the user's specific criteria. X-Class addresses this by generating class-oriented representations that focus on features relevant to the provided class names.

The X-Class Framework

The X-Class framework operates in four distinct stages to bridge the gap between raw text and a trained classifier.

Overview of the X-Class framework pipeline.

1. Class Representation Estimation

The process begins by estimating a representation for each class. Starting with the user-provided class name, the system iteratively expands the class's "vocabulary" by adding the most semantically similar words from the corpus. To prevent semantic drift, a consistency check is employed to stop the expansion before the meaning of the class becomes diluted.

2. Class-Oriented Document Representation

Instead of treating a document as a static vector, X-Class uses a mixture of attention mechanisms to create an adaptive representation. It calculates the relevance of each word in a document to the specific class names. This "steers" the document's representation to highlight features relevant to the classification task—for instance, focusing on geographical terms when the classes are locations.

The class-oriented attention mechanism.

3. Document-Class Alignment

Once documents are represented in a class-oriented space, they are clustered using a Gaussian Mixture Model (GMM). Unlike K-Means, which assumes spherical clusters, GMM accounts for the covariance of the data, allowing for elliptical shapes that better represent the distribution of natural language embeddings.

4. Classifier Training

The system identifies the most "confident" documents within each GMM cluster—those with the highest probability of belonging to a specific class. These documents serve as a pseudo-labeled training set used to fine-tune a final supervised classifier, such as BERT.

Experimental Evaluation

X-Class was evaluated across seven diverse benchmark datasets covering various classification criteria, including topic-based (AGNews, 20News, NYT-Topic), location-based (NYT-Location), and sentiment-based (Yelp) tasks.

Key Results

  • Superiority over Seed-Driven Methods: X-Class frequently outperformed methods like WeSTClass and ConWea, which require multiple seed words per class, demonstrating that well-estimated class representations can be more effective than manually picked seeds.
  • Robustness to Imbalance: On the NYT-Small dataset, X-Class achieved a Micro-F1 score of 97.18, significantly outperforming LOTClass (78.12), another class-name-only method that struggles with class imbalance.
  • The "Power of One": Experiments showed that X-Class is remarkably robust to class name frequency. Even when a class name appeared only once in the entire corpus, performance dropped by less than 1%.

Discussion and Insights

The success of X-Class highlights that document representations must be adaptive. The same news article can be represented differently depending on whether the goal is to classify it by topic, location, or sentiment. By making the embedding process "class-oriented," X-Class captures the nuances required for diverse tasks. Furthermore, the choice of GMM over K-Means proves vital, as document clusters in embedding space are rarely perfectly spherical; capturing their "stretch" and "tilt" is essential for accurate alignment.

Conclusion

X-Class demonstrates that extremely weak supervision is a viable and powerful alternative to traditional labeling. By leveraging class-oriented attention and robust clustering, it minimizes human effort while maintaining high performance across varied classification tasks. Future work may extend this approach to more complex NLP tasks like Named Entity Recognition and Entity Linking.

Summary updated

BibTeX
@inproceedings{wang2021x,
  author = {Zihan Wang and Dheeraj Mekala and Jingbo Shang},
  title = {{X-Class: Text Classification with Extremely Weak Supervision}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=E-Zg-ltT0Fm},
}
Figure