SALNet: Semi-supervised Few-Shot Text Classification with Attention-based Lexicon Construction

Ju Hyoung Lee, Sang-Ki Ko, Yo-Sub HanAAAI 2021 · DOI

Abstract

SALNet is a semi-supervised framework designed to address the "data hunger" of deep learning text classifiers. By leveraging attention mechanisms to automatically construct class-specific lexicons, SALNet enhances pseudo-labeling accuracy and mitigates error accumulation in few-shot learning. Experimental results across four benchmark datasets demonstrate that SALNet significantly outperforms standard bootstrapping methods and competitive baselines like fine-tuned BERT in low-resource environments.

Introduction

Deep learning has revolutionized text classification, yet these models typically require thousands of labeled examples to achieve high performance. In many real-world scenarios, manual annotation is prohibitively expensive, leading to a reliance on few-shot learning. Standard Semi-Supervised Learning (SSL) attempts to bridge this gap by using a small labeled set to "pseudo-label" a larger unlabeled dataset. However, weak initial models often introduce errors that accumulate during iterative training, causing performance to plateau or even drop.

The SALNet Framework

To overcome the limitations of standard SSL, the authors introduce SALNet (Semi-supervised with Attention-based Lexicon construction Network). This framework integrates traditional lexicon-based robustness with modern neural network capabilities through an iterative bootstrapping process.

The SALNet framework overview, illustrating the iterative process of training, lexicon generation, and dual-criteria pseudo-labeling.

The core innovation lies in using the attention weights of an initial LSTM-based classifier to identify class-specific keywords. Even when a model is uncertain about a full sentence, its attention mechanism often correctly identifies salient words (e.g., "stupid" in a negative movie review). These words are aggregated into a lexicon for each class, turning the "black box" of the neural network into a transparent list of keywords.

Dual-Criteria Pseudo-Labeling

SALNet employs a sophisticated dual-criteria strategy to label unlabeled data and filter noise:

  1. The Double Check: A label is accepted only if the classifier is highly confident and the text contains a minimum threshold of words from the corresponding lexicon.
  2. The Rescue: If the classifier has low confidence, the lexicon can "override" the model and assign a label if sufficient keyword matches are present. This prevents the model from ignoring useful unlabeled data just because the initial neural weights were poorly initialized.

Experimental Results

SALNet was evaluated on four benchmark datasets: IMDB (sentiment), AG News (news topics), Yahoo! Answers (QA topics), and DBpedia (Wikipedia topics). To simulate few-shot conditions, the researchers used only 0.2% and 1% of the original training data for the initial supervised phase.

Performance comparison of SALNet against baselines and other SSL methods using 1% of labeled data.

As shown in the results for the 1% data setting, SALNet consistently outperformed standard SSL methods. Notably, on the IMDB dataset, SALNet achieved an accuracy of 84.87%, compared to 74.97% for standard self-training. In the even more restrictive 0.2% setting, SALNet frequently outperformed fine-tuned BERT, suggesting that specialized lexicon construction is more effective than general-purpose pre-training when data is extremely scarce.

Conclusion

SALNet demonstrates that attention mechanisms can be effectively repurposed as knowledge extractors to automate lexicon construction. By requiring "symbolic evidence" from these lexicons before trusting a pseudo-label, the system remains robust against the noise that typically plagues self-training. Future work will focus on addressing word ambiguity across different classes and integrating more advanced pre-trained models for initial lexicon generation.

Summary updated

BibTeX
@inproceedings{lee2021salnet,
  author = {Ju Hyoung Lee and Sang-Ki Ko and Yo-Sub Han},
  title = {{SALNet: Semi-supervised Few-Shot Text Classification with Attention-based Lexicon Construction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i14.17558},
}
Figure