Abstract
Standard text classification models are typically restricted to a fixed set of labels, struggling to generalize to novel categories encountered in real-world applications. This paper addresses this limitation by introducing two novel pre-training strategies—Implicit and Explicit pre-training—that focus on "aspect-level" understanding rather than label-specific memorization. By teaching models to recognize the broader context of a task (such as sentiment, intent, or topic), the researchers achieved significant improvements in zero-shot generalization. Evaluated on a new 18-dataset benchmark called UTCD, these methods demonstrate up to a 9% improvement in accuracy, outperforming much larger models like GPT-3 in several out-of-domain scenarios.
The Challenge of "Label-Locked" Models
Traditional text classification models are often "label-locked." A model trained to distinguish between "Happy" and "Sad" will likely fail when asked to identify a new category like "Frustrated." While Zero-shot Learning (ZSL) aims to bridge this gap, current Pre-trained Language Models (PLMs) often lack the necessary aspect-level understanding to generalize effectively. They may learn the specific features of a "Refund" label in a banking context but fail to grasp the underlying concept of "Intent Classification."
Proposed Methods
The authors propose injecting aspect-level knowledge into models through two distinct strategies applied across three common architectures: Binary BERT (cross-attention), Bi-Encoders (Dual Encoding), and Generative Models (GPT-style).
1. Implicit Pre-training
Inspired by the T5 framework, this method adds a conditional "aspect token" (e.g., [sentiment], [intent], or [topic]) to the input. This forces the model to learn contextual representations for text-label pairs conditioned on the specific task type, effectively providing a lens through which the model interprets the data.
2. Explicit Pre-training
This strategy utilizes a two-step initialization process. The model is first trained on a "warm-up" task called Aspect Detection, where it learns to distinguish high-level categories (sentiment vs. intent vs. topic). Once the internal representations are tuned to these aspects, the model is fine-tuned for the final zero-shot classification objective.
The UTCD Benchmark
To rigorously evaluate these methods, the authors developed the Universal Text Classification Dataset (UTCD). This benchmark compiles 18 existing datasets structured around three main aspects: Sentiment, Intent/Dialogue, and Topic. The data is split into "in-domain" sets for training and "out-of-domain" sets (containing completely unseen labels and datasets) for evaluation, ensuring a true test of zero-shot generalization.
Key Insights and Results
- Significant Performance Gains: Explicit pre-training on a Binary BERT model resulted in a 9% average improvement in accuracy across all out-of-domain tasks. In the "Topic" aspect, the improvement reached as high as +16%.
- Efficiency vs. Scale: The tuned models (based on BERT and GPT-2) frequently outperformed significantly larger zero-shot classifiers, including BART-MNLI and GPT-3, on 7 out of 9 out-of-domain datasets. This suggests that targeted pre-training can allow smaller models to compete with massive ones.
- The Importance of Task Context: Simply providing label names is insufficient; providing the model with the type of task (the aspect) is crucial for bridging the gap between seen and unseen concepts.
- Label Overlap: Success in zero-shot classification is highly correlated with "token overlap." Models perform better when unseen labels share vocabulary with labels encountered during training, suggesting that models anchor new concepts to familiar linguistic foundations.
Limitations and Future Directions
While effective, the current study is limited to English-language datasets and three broad aspects. Future work will explore scaling these strategies to dozens or hundreds of more granular aspects and extending the methodology to multi-lingual contexts to better cover the vast space of text interpretation.
Summary updated