TL;DR
This paper tackles the challenge of few-shot text classification where semantically similar labels can confuse models. They introduce a novel method that enhances label representations by explicitly learning to distinguish between them, leading to significant performance improvements across various few-shot classification models.
What Problem Does This Paper Address?
Few-shot text classification is a crucial task where models must learn to classify text with very limited examples for each label. A significant hurdle in this area is when labels are semantically similar. Traditional methods often struggle to differentiate between these closely related labels, leading to misclassifications. This paper highlights that existing approaches overlook the semantic relevance between label representations, causing confusion and hindering performance, especially when dealing with skewed label distributions.
Methods
The authors propose a method that augments existing few-shot classification models with a "difference extractor" and a novel loss function based on mutual information.
- Difference Extractor: This component takes the initial label representations (generated by a standard encoder and pooling function) and a query representation. It uses a multi-head self-attention mechanism to process these inputs simultaneously, generating refined label representations that capture information specific to each label. These refined representations are then passed through fully-connected layers with a GELU activation.
- Mutual Information (MI) Loss Function: The core idea here is to ensure that each refined label representation is independent of others, meaning it only contains information specific to its own label. They aim to minimize the mutual information between different label representations. Since directly calculating MI is difficult, they minimize an upper bound of it.
- Overall Loss: The final loss function is a combination of the standard classification loss (negative log-likelihood) and the proposed MI-based loss, with a hyperparameter
αto balance their contributions.
This approach is designed to be modular and can be applied to various conventional few-shot classification models.
Datasets
The paper evaluates their method on two standard benchmarks for few-shot text classification:
- Huffpost: This dataset consists of article titles from HuffPost, with the task of predicting an article's category. It has 41 label types in the training set, 5 in validation, and 16 in the test set, with 900 examples per label.
- FewRel: This dataset focuses on relation extraction between entities. It features 65 label types in the training set, 5 in validation, and 10 in the test set, with 700 examples per label.
Key Insights and Ideas
- Semantic Similarity is a Major Bottleneck: The paper strongly emphasizes that the semantic relatedness of labels is a critical, often overlooked, factor limiting few-shot classification performance.
- Explicitly Learning Differences is Key: Instead of relying solely on the inherent discriminative power of embeddings, the proposed method actively learns to create distinct representations for each label.
- Mutual Information as a Regularizer: Using mutual information to enforce independence between label representations is an elegant way to ensure that each representation captures unique information, preventing confusion.
- Modularity is a Strength: The ability to easily integrate the proposed difference extractor and MI-loss into existing models (like ProtoNet, MAML, and MLMAN) makes it a practical and broadly applicable solution.
Key Results
The proposed method, denoted as "+ DE + Î", consistently improved the performance of baseline few-shot classification models across both Huffpost and FewRel datasets.
- Significant Accuracy Gains: For MAML and MLMAN, the full method consistently improved performance, with gains ranging from 0.61 to 7.68 points.
- ProtoNet Benefits Most: ProtoNet, with its simpler architecture, showed substantial improvements, ranging from 0.39 to 2.13 points when using both the difference extractor and MI-loss. This highlights the importance of both components for less complex models.
- Ablation Study: Applying only the difference extractor ("+ DE") sometimes led to a slight performance degradation on FewRel for ProtoNet, suggesting that the MI-loss is crucial for this model.
- State-of-the-Art Performance: The paper reports achieving state-of-the-art accuracy on the Huffpost dataset.
Limitations and Future Work
- Hyperparameter Sensitivity: The paper notes that setting the
αhyperparameter too high can lead to performance degradation. This is attributed to the model extracting spurious differences that are irrelevant to the classification task. - Generalizability to Other Domains: While tested on text classification, the authors suggest that the method could be applied to other classification problems involving semantic labels, such as image and entity classifications. Future work could involve evaluating its effectiveness in these broader domains.
Summary updated