Extremely Weakly-supervised Text Classification with Wordsets Mining and Sync-Denoising

Lysa XiaoNAACL 2024 · DOI

TL;DR

This paper introduces SetSync, a novel framework for extremely weakly-supervised text classification (WTC) that leverages "wordsets" to represent categories, overcoming limitations of existing prompt- and seed-based methods. SetSync also proposes a new "sync-denoising" training strategy to effectively handle noisy pseudo-labels and unlabeled data, achieving state-of-the-art results across 11 datasets.

What Problem Does This Paper Address?

Extremely weakly-supervised text classification (WTC) aims to classify text using only class names as supervision, avoiding costly manual annotation. Existing methods fall into two main categories: prompt-based methods, which use language models (like GPT) with instructions, and seed-based methods, which expand class names into related words for matching. However, prompt-based methods suffer from zero-shot instability and context-dependent ambiguities, while seed-based methods can misclassify text due to keyword ambiguity (e.g., "not bad" being misclassified as negative). Furthermore, existing WTC approaches often overlook the noise inherent in the generated pseudo-labels, leading to error propagation during classifier training, especially when using self-training.

Methods

SetSync tackles these challenges with a two-pronged approach:

  • Wordset-based Pseudo-Label Generation: Instead of individual keywords, SetSync represents categories with "wordsets" – collections of related keywords. This helps capture more nuanced meanings and avoid ambiguity. To mine these class-relevant wordsets, the paper proposes "Wordsets Information Bottleneck" (WIB), inspired by information bottleneck theory. WIB aims to find compressed wordset representations that retain maximum information about the class. The process involves high-frequency screening of words, frequent itemset mining (using FP-Growth), and then ranking wordsets based on their mutual information with the class.
  • Sync-Denoising Training: Recognizing that WTC data is a hybrid of semi-supervised (few noisy pseudo-labels) and unsupervised (unlabeled data) learning, SetSync introduces a novel "sync-denoising" training strategy. This strategy jointly optimizes both pseudo-labeled and unlabeled data within a unified denoising framework. It dynamically samples weights for each data type based on estimated noise levels, derived from loss values for pseudo-labeled data and prediction confidence for unlabeled data, using dynamic Gaussian distributions.

The framework operates iteratively: initial wordsets are derived from class names, pseudo-labels are generated, the classifier is trained using sync-denoising, and then wordsets are mined and updated for the next iteration.

Datasets

The paper evaluates SetSync on 11 text datasets, following a recent WTC benchmark. These datasets span various domains and tasks:

  • Sentiment Analysis: IMDB, Yelp-2, Yelp-5
  • Topic Classification: AGNews, 20News, 20News-Fine
  • Entity Recognition and Classification: DBpedia
  • News Text and Fine-grained Analysis: NYT, NYT-Fine, NYT-Topics, NYT-Loc

Key Insights and Ideas

  • Wordsets over Keywords: Representing categories with wordsets (e.g., {"not", "bad"}) is more robust to ambiguity and captures richer semantic meaning than single keywords (e.g., "bad").
  • Information Bottleneck for Wordset Mining: Applying information bottleneck principles to wordset mining effectively balances informativeness and compression, leading to better class-relevant wordsets.
  • Unified Denoising for Noisy and Unlabeled Data: Treating WTC as a combined noisy-label and semi-supervised learning problem and developing a unified sync-denoising strategy is crucial for effective classifier training.
  • Dynamic Noise Estimation: Dynamically estimating noise levels using both loss values (for pseudo-labels) and prediction confidence (for unlabeled data) is more effective than relying on a single metric.

Key Results

SetSync achieved state-of-the-art performance across all 11 datasets, outperforming existing seed-based and prompt-based methods by an average of 8 points in accuracy. For instance, with BERT-base, SetSync surpassed ClassKG by 9.0 points and the best prompt method (Prompt+DCPMI) by 24.70 points. The sync-denoising training strategy consistently outperformed other strategies like fully-supervised training or standard self-training. Ablation studies confirmed the effectiveness of both the wordset mining and sync-denoising components.

Limitations and Future Work

The authors acknowledge that SetSync's running time might be longer than seed methods due to the frequent itemset mining process, although high-frequency filtering helps accelerate it. The iterative nature of the method also contributes to its runtime. Future work could explore more efficient mining techniques. The paper also notes that while SetSync can surpass ChatGPT-3.5 on some datasets, ChatGPT-3.5 achieved higher results on others, suggesting potential for further integration or comparison with large language models.

Summary updated

BibTeX
@inproceedings{xiao2024extremely,
  author = {Lysa Xiao},
  title = {{Extremely Weakly-supervised Text Classification with Wordsets Mining and Sync-Denoising}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.397},
}
Figure