TL;DR
This paper introduces T5 Chain, a novel approach for few-shot multi-label text classification (MLTC) that leverages the power of T5's text-to-text capabilities. By treating MLTC as a sequence-to-set prediction problem and employing a three-stage training strategy, T5 Chain effectively models label dependencies while maintaining efficient inference, outperforming existing methods on several benchmarks.
What Problem Does This Paper Address?
Multi-label text classification (MLTC) is a crucial task in NLP, where a document can belong to multiple categories simultaneously. Traditional MLTC methods often require extensive labeled data, which is expensive and time-consuming to obtain, especially for new domains. While large language models (LLMs) offer promise for zero-shot and few-shot learning, they can be computationally expensive to deploy and sensitive to prompting. Existing few-shot MLTC approaches often rely on training multiple independent classifiers (e.g., one-vs-all), which fail to capture inter-label dependencies and require retraining for new label sets. Smaller encoder-decoder models like T5 can generate labels sequentially, potentially capturing these dependencies, but earlier attempts suffered from order bias and didn't fully utilize label semantics. This paper aims to bridge this gap by developing an efficient and effective few-shot MLTC method that leverages label semantics and dependencies without the drawbacks of previous approaches.
Methods
The core of this work is the T5 Chain model, which frames MLTC as a sequence-to-set prediction task using a T5 encoder-decoder architecture. The input to the model is a concatenation of the text and its candidate labels, demarcated by special sentinel tokens (e.g., [E1], [E2]). The T5 decoder then generates a sequence of "YES" or "NO" tokens corresponding to each label.
The training process is divided into three stages:
- Generalist Pre-training (GP): This stage aims to teach the T5 model to classify topics in an order-agnostic manner. It uses a large, diverse dataset (The Pile, augmented with GPT-40 for topic extraction) and a permutation-SGD objective to learn general topic classification without being tied to a specific label order. This stage imbues the model with foundational knowledge.
- Order-Marginal Domain Adaptation (DA): Here, the model is fine-tuned on domain-specific data using a standard cross-entropy loss. Crucially, label shuffling is employed during this stage to maintain permutation robustness and adapt the model to the target domain's topics.
- Chain Specialization (CS): In the final stage, a single, optimal label chain (permutation) is selected based on validation performance. The model is then further fine-tuned on this fixed chain. This specialization step refines conditional dependencies between labels while ensuring efficient, single-sequence inference.
Datasets
The paper utilizes the following datasets:
- The Pile: A large-scale, diverse corpus used for generalist pre-training.
- GPT-40: Used to generate topic labels for texts from The Pile, creating a distilled dataset for pre-training.
- Eurlex: A dataset of EU legislation documents, using top-level labels from its hierarchy.
- AAPD (ArXiv Academic Paper Dataset): Contains abstracts from arXiv academic papers, tagged with arXiv subject categories.
- Reuters-21578: A popular dataset from the Reuters financial newswire service.
The paper does not introduce new benchmarks but evaluates on existing ones to demonstrate its effectiveness.
Key Insights and Ideas
- Text-to-Text for MLTC: Framing MLTC as a text-to-text generation problem with T5 is a powerful paradigm, allowing for unified handling of text and labels.
- Order-Agnostic Pre-training is Crucial: The Generalist Pre-training (GP) stage is vital. Removing it leads to a significant performance drop, highlighting the importance of foundational knowledge from diverse data for few-shot downstream tasks.
- Balancing Order-Invariance and Efficiency: T5 Chain strikes a balance between the theoretical benefits of order-invariant methods and the practical efficiency of single-chain inference.
- Label Semantics Matter: By encoding label semantics within the input sequence, the model can better understand and utilize them, especially for rare labels.
- Label Dependency is Dataset-Dependent: The benefit of modeling label dependencies is not universal. Datasets with sparse label correlations (like Reuters) show less improvement from classifier chains compared to those with richer correlations (Eurlex, AAPD).
Key Results
- State-of-the-Art Few-Shot Performance: T5 Chain consistently outperforms baselines across three few-shot MLTC datasets (Eurlex, AAPD, Reuters) in terms of micro-F1 (mF1), macro-F1 (MF1), and instance-F1 (IF1) scores.
- Significant Gains on Eurlex and AAPD: On Eurlex, T5 Chain improves mF1 and IF1 by approximately 3 points over PET. On AAPD, it surpasses SetFit by about 2 points.
- Strong Performance on Reuters: T5 Chain achieves the best MF1 and IF1 scores on Reuters (57.7 and 88.9, respectively), demonstrating good global performance and robustness to rare labels.
- Robust Macro-Level Performance: T5 Chain shows particularly strong macro-level performance, exceeding other methods by over 10 points in MF1 across datasets, which is important for imbalanced label distributions.
- Stable Performance: The method exhibits stable performance across different few-shot splits, with relative deviations below 3.5%.
- Zero-Shot Performance: T5 Generalist (before chain specialization) shows strong zero-shot performance against NSP and PET.
Limitations and Future Work
- Inference Latency: Sequential label prediction introduces inference latency, especially for long label sequences. Chunking label sequences might be necessary but could compromise global dependency modeling.
- Optimal Chain Selection: The current method relies on empirical evaluation to select the optimal label order. Developing a more generalized and reliable method for chain selection, particularly with limited supervised data, is a direction for future research.
- Distillation Data Quality: The performance of the pre-trained model depends on the quality of the distillation data. Enhancing or denoising labels from LLMs could potentially boost downstream generalization.
- Dataset-Specific Benefits: The paper notes that the benefits of modeling label dependencies are less pronounced on datasets with sparse label correlations, suggesting that the approach might not be universally advantageous for all MLTC scenarios.
Summary updated