X-Class: Text Classification with Extremely Weak Supervision

Zihan Wang, Dheeraj Mekala, Jingbo Shang

Abstract

In this paper, we explore text classification with extremely weak supervision, i.e., only relying on the surface text of class names. This is a more challenging setting than the seed-driven weak supervision, which allows a few seed words per class. We opt to attack this problem from a representation learning perspective -- ideal document representations should lead to nearly the same results between clustering and the desired classification. In particular, one can classify the same corpus differently (e.g., based on topics and locations), so document representations should be adaptive to the given class names. We propose a novel framework X-Class to realize the adaptive representations. Specifically, we first estimate class representations by incrementally adding the most similar word to each class until inconsistency arises. Following a tailored mixture of class attention mechanisms, we obtain the document representation via a weighted average of contextualized word representations. With the prior of each document assigned to its nearest class, we then cluster and align the documents to classes. Finally, we pick the most confident documents from each cluster to train a text classifier. Extensive experiments demonstrate that X-Class can rival and even outperform seed-driven weakly supervised methods on 7 benchmark datasets.

Introduction

Text classification is a cornerstone of NLP, but training a classifier usually requires a mountain of labeled data. "Weak supervision" emerged to solve this by using a few "seed words" per class to bootstrap the process. However, even picking 3–5 perfect seed words requires a human expert to deeply understand the corpus.

The authors tackle extremely weak supervision, where the only input is the class name itself. The core challenge here is that a single document can be classified in multiple ways. A news article about a basketball game in Los Angeles could be categorized by Topic (Sports) or Location (USA). Standard text embeddings (like those from BERT) tend to cluster by topic by default. If a user wants to classify by location or sentiment, standard "one-size-fits-all" representations fail. X-Class aims to make document representations adaptive to the user's chosen class names.

The X-Class Framework

X-Class follows a three-stage pipeline to turn raw text and class names into a high-performing classifier:

Overview of the X-Class framework pipeline.

1. Class-Oriented Representation Estimation

The system starts with the class name and "expands" it by iteratively adding the most similar words found in the corpus. Crucially, it uses a consistency check: it stops adding words if the new words start shifting the meaning of the class too far away from the original name. This prevents "topic drift."

2. Document-Class Alignment

Instead of a simple average of word vectors, X-Class uses a mixture of attention mechanisms. It weights words based on how well they align with the estimated class representations. It then uses Gaussian Mixture Models (GMM) to cluster the documents. Unlike K-Means, GMM allows for elliptical cluster shapes, which better fits the reality of how document embeddings are distributed.

3. Text Classifier Training

The clustering step provides "pseudo-labels." X-Class selects the most confident documents (the top 50%) from each cluster and uses them to fine-tune a final BERT classifier. This final step helps the model generalize to unseen data and smooths out the noise from the clustering phase.

Key Insights: Overcoming Topic Bias

One of the most striking findings is that standard BERT representations are excellent at separating topics but often fail at separating other criteria like locations. This proves that "unsupervised" classification isn't enough; the model must be guided by the class names to know which features of the text matter.

Visualization of how document representations can be clustered differently based on the target classification criteria.

X-Class's Adaptive Keyword Expansion recognizes that some classes are "narrower" than others. Its iterative expansion with a consistency check allows the model to be flexible—using many words for a broad class like "Sports" and fewer for a specific one. Furthermore, X-Class does not require the class name to appear frequently in the corpus; performance remains stable even if the class name occurs only once.

Experimental Results

The authors evaluated X-Class on 7 benchmark datasets covering diverse domains, including Topic Classification (AGNews, 20News, NYT-Topic, DBpedia), Location Classification (NYT-Location), and Sentiment Analysis (Yelp reviews).

Despite using only the class name, X-Class outperformed methods like WeSTClass and ConWea, which were given 3+ expert-chosen seed words per class. On the NYT-Location dataset, X-Class achieved a 91.8 F1 score, vastly outperforming LOTClass (58.5) and WeSTClass (63.2). This highlights the success of its adaptive representation strategy.

Conclusion

X-Class demonstrates that extremely weak supervision is a viable path for text classification, matching or exceeding the performance of more labor-intensive methods. By making document representations adaptive to specific class names, it overcomes the inherent biases of pre-trained language models. Future work may extend this approach to tasks like Named Entity Recognition (NER) and Entity Linking.

Dataset and code are released at https://github.com/ZihanWangKi/XClass/ .

Summary updated

Abstract

In this paper, we explore text classification with extremely weak supervision, i.e., only relying on the surface text of class names. This is a more challenging setting than the seed-driven weak supervision, which allows a few seed words per class. We opt to attack this problem from a representation learning perspective -- ideal document representations should lead to nearly the same results between clustering and the desired classification. In particular, one can classify the same corpus differently (e.g., based on topics and locations), so document representations should be adaptive to the given class names. We propose a novel framework X-Class to realize the adaptive representations. Specifically, we first estimate class representations by incrementally adding the most similar word to each class until inconsistency arises. Following a tailored mixture of class attention mechanisms, we obtain the document representation via a weighted average of contextualized word representations. With the prior of each document assigned to its nearest class, we then cluster and align the documents to classes. Finally, we pick the most confident documents from each cluster to train a text classifier. Extensive experiments demonstrate that X-Class can rival and even outperform seed-driven weakly supervised methods on 7 benchmark datasets. Our dataset and code are released at https://github.com/ZihanWangKi/XClass/ .
BibTeX
@inproceedings{wang2021x,
  author = {Zihan Wang and Dheeraj Mekala and Jingbo Shang},
  title = {{X-Class: Text Classification with Extremely Weak Supervision}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=E-Zg-ltT0Fm},
}
Figure