Unsupervised Label Refinement Improves Dataless Text Classification

Zewei Chu, Karl Stratos, Kevin GimpelACL 2021

TL;DR

This paper introduces Unsupervised Label Refinement (ULR), a novel clustering-based approach that significantly improves dataless text classification by leveraging unlabeled data to refine label representations. ULR makes dataless classifiers more robust to variations in label descriptions, a critical limitation that hinders their practical application.

What Problem Does This Paper Address?

Dataless text classification aims to categorize text into labels without any task-specific labeled data. This is achieved by using a classifier that can score any document-label pair, often relying on descriptive text for each label. While powerful, this approach is highly sensitive to the quality of these label descriptions. Poorly chosen descriptions can drastically reduce classification accuracy, making dataless classifiers fragile and limiting their real-world utility. The authors address this by asking: how can we improve dataless text classification by using the unlabeled text data from the downstream task itself, in addition to the label descriptions?

Methods

The core of the paper is the Unsupervised Label Refinement (ULR) algorithm. ULR leverages the assumption that documents belonging to the same category should be close in the encoded semantic space. It refines the classifier's label representations using a k-means clustering approach.

The ULR algorithm has variations depending on the underlying dataless classifier architecture:

  • Dual Encoder Architectures: ULR performs k-means clustering on the text representations (vectors produced by the text encoder). The centroids are initialized using the category encodings and are then iteratively updated based on text assignments. A key innovation here is the interpolation of new centroids with the original category embeddings, acting as a form of regularization.
  • Single Encoder Architectures: For single encoder models, ULR clusters the probability distributions of documents over categories. The centroids are initialized as one-hot vectors representing categories, and the algorithm aims to find soft distributions for categories.

The paper also explores variations like early stopping and centroid interpolation for the single encoder model, though these did not yield superior results.

Datasets

The paper evaluates ULR on four standard text classification datasets:

  • AG News (AG): A 4-class dataset covering the newswire domain.
  • DBpedia (DBP): A 14-class dataset from the encyclopedia domain.
  • Yahoo: A 10-class dataset for categorizing online question forum content.
  • 20 Newsgroups (20NG): A 20-class dataset representing different newsgroup topics.

The authors also mention using the NATCAT dataset for fine-tuning ROBERTA models, which consists of document-category pairs with positive and negative labels from Wikipedia, Stack Exchange, and Reddit.

Key Insights and Ideas

  • Leveraging Unlabeled Data for Robustness: The most significant insight is that unlabeled text data, often readily available in a downstream task, can be used to significantly improve the robustness of dataless classifiers against suboptimal label descriptions.
  • Clustering as a Refinement Mechanism: ULR demonstrates that applying clustering to document representations (or probability distributions) can effectively refine the semantic understanding of labels, bringing them closer to the actual data distribution.
  • Interpolation as Regularization: The interpolation of updated centroids with initial category embeddings in the dual encoder ULR is a clever regularization technique that helps stabilize the clustering process and leverage prior knowledge from the dataless classifier.
  • ULR's Broad Applicability: The method shows consistent improvements across different dataless classifier architectures (GloVe, ROBERTA dual/single encoders) and various distance/scoring functions, highlighting its general utility.

Key Results

  • Consistent Accuracy Improvements: ULR consistently improves accuracy across all tested datasets and classifier architectures. For instance, with the ROBERTA dual encoder, ULR improves accuracy by 3.6% to 4.4% on average. For the single encoder, improvements range from 0.5% to 6.8%.
  • Mitigating Label Noise: Experiments with varying label names show that ULR significantly improves performance in the vast majority of cases, even when label names are suboptimal. For example, in robustness experiments on AG News, ULR improved accuracy in 99.6% of cases with cosine distance.
  • Ensemble Benefits: Combining ULR with label ensembles (using multiple sets of label names) further boosts performance, demonstrating that ULR can enhance even strong baseline configurations.
  • Performance Correlation: ULR tends to provide larger accuracy gains when the initial dataless classifier's accuracy is lower, suggesting it's particularly effective in challenging scenarios.

Limitations and Future Work

  • Computational Cost: While ULR improves performance, it adds computational overhead due to the clustering step. The paper mentions that ESA vectors are computationally difficult to apply ULR to, suggesting that vector dimensionality is a factor.
  • Initialization Sensitivity: While ULR uses category embeddings for initialization, the paper briefly explores random initialization for the single encoder, noting that while it improves accuracy, it loses the direct link between centroids and categories.
  • Future Work: The paper suggests that ULR is orthogonal to methods that generate better label names, implying that these could be combined. They also briefly touch upon few-shot learning scenarios, where ULR can further enhance performance when combined with a small number of labeled instances. The robustness experiments also hint at future work in mitigating harmful effects of suboptimal label names.

Summary updated

BibTeX
@inproceedings{chu2021unsupervised,
  author = {Zewei Chu and Karl Stratos and Kevin Gimpel},
  title = {{Unsupervised Label Refinement Improves Dataless Text Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure