Abstract
Long-tailed relation classification is a challenging problem where head classes dominate the training phase, leading to poor performance on tail classes. Existing solutions often entangle representation learning and classifier training. In this study, we conduct an empirical investigation and find that pre-trained models with instance-balanced sampling already capture well-learned representations for all classes. We demonstrate that better long-tailed classification can be achieved at a low cost by only adjusting the classifier. We propose a robust classifier with Attentive Relation Routing (ARR), which uses dynamic routing to aggregate relations. Extensive experiments on two datasets demonstrate the effectiveness of our approach.
Introduction
Relation Classification (RC) involves identifying the relationship between entities in text. In real-world scenarios, these relations often follow a long-tailed distribution: a few "head" relations have many examples, while many "tail" relations have very few. Traditional methods address this via re-sampling or re-weighting but fail to distinguish whether the bottleneck lies in the learned features or the classifier's decision boundaries.
Decoupling Representation and Classification
To diagnose the root cause of poor tail performance, we employ a decoupling analysis. Training is split into two stages:
- Representation Learning: A feature extractor (e.g., BERT) is trained using various strategies like Instance-Balanced Sampling (IBS), Class-Balanced Sampling (CBS), or Re-Weighting Loss (RWL).
- Classifier Learning: The feature extractor is frozen, and only the classifier is retrained.
Our findings reveal that "The Devil is the Classifier." Pre-trained models like BERT, when trained on standard instance-balanced data, actually learn high-quality representations for all classes. The performance drop on tail classes occurs because the classifier's weights become biased toward the majority classes. Interestingly, re-sampling strategies (CBS) during the first stage can sometimes degrade the quality of the learned features, even if they help the classifier.
Attentive Relation Routing (ARR)
Based on these insights, we propose Attentive Relation Routing (ARR). Inspired by Capsule Networks, ARR replaces the standard linear classifier with a dynamic routing mechanism. Instead of fixed "hard" weights, ARR treats sentence features as "primary capsules" and iteratively routes them to "relation capsules" using soft weights. This allows the model to more flexibly aggregate relevant features, which is particularly beneficial for tail classes that may share patterns with head classes.
Key Results and Insights
The ARR method consistently outperforms baseline models and traditional re-balancing techniques across benchmarks like TACRED and FewRel-LT.
- Balanced Improvement: Unlike many methods that improve tail performance at the expense of the head, ARR improves scores across the entire distribution. On FewRel-LT, it achieves an improvement of approximately 5% in Macro F1 score.
- Efficiency: Significant gains are achieved by simply optimizing the classifier architecture, avoiding the high cost of retraining large language models with complex sampling schemes.
- Robustness: The iterative routing process in ARR provides a more robust decision boundary, effectively mitigating the bias inherent in long-tailed data.
Conclusion
Our study shifts the focus from representation learning to classifier design in long-tailed relation classification. By decoupling these two stages, we identified the classifier as the primary source of bias. The proposed ARR mechanism offers a cost-effective and high-performance solution for extracting relations from imbalanced data.
Summary updated