The Devil is the Classifier: Investigating Long Tail Relation Classification with Decoupling Analysis

Haiyang Yu, Ningyu Zhang, Shumin Deng, Zonggang Yuan, Yantao Jia, Huajun ChenarXiv 2020 · OpenReview

Abstract

Long-tailed relation classification is a challenging problem where head classes dominate the training phase, leading to poor performance on tail classes. Existing solutions often entangle representation learning and classifier training. In this study, we conduct an empirical investigation and find that pre-trained models with instance-balanced sampling already capture well-learned representations for all classes. We demonstrate that better long-tailed classification can be achieved at a low cost by only adjusting the classifier. We propose a robust classifier with Attentive Relation Routing (ARR), which uses dynamic routing to aggregate relations. Extensive experiments on two datasets demonstrate the effectiveness of our approach.

Introduction

Relation Classification (RC) involves identifying the relationship between entities in text. In real-world scenarios, these relations often follow a long-tailed distribution: a few "head" relations have many examples, while many "tail" relations have very few. Traditional methods address this via re-sampling or re-weighting but fail to distinguish whether the bottleneck lies in the learned features or the classifier's decision boundaries.

Decoupling Representation and Classification

To diagnose the root cause of poor tail performance, we employ a decoupling analysis. Training is split into two stages:

  1. Representation Learning: A feature extractor (e.g., BERT) is trained using various strategies like Instance-Balanced Sampling (IBS), Class-Balanced Sampling (CBS), or Re-Weighting Loss (RWL).
  2. Classifier Learning: The feature extractor is frozen, and only the classifier is retrained.
Decoupling analysis of representation and classifier learning.

Our findings reveal that "The Devil is the Classifier." Pre-trained models like BERT, when trained on standard instance-balanced data, actually learn high-quality representations for all classes. The performance drop on tail classes occurs because the classifier's weights become biased toward the majority classes. Interestingly, re-sampling strategies (CBS) during the first stage can sometimes degrade the quality of the learned features, even if they help the classifier.

Attentive Relation Routing (ARR)

Based on these insights, we propose Attentive Relation Routing (ARR). Inspired by Capsule Networks, ARR replaces the standard linear classifier with a dynamic routing mechanism. Instead of fixed "hard" weights, ARR treats sentence features as "primary capsules" and iteratively routes them to "relation capsules" using soft weights. This allows the model to more flexibly aggregate relevant features, which is particularly beneficial for tail classes that may share patterns with head classes.

Key Results and Insights

The ARR method consistently outperforms baseline models and traditional re-balancing techniques across benchmarks like TACRED and FewRel-LT.

Performance comparison across head, medium, and tail relation buckets.
  • Balanced Improvement: Unlike many methods that improve tail performance at the expense of the head, ARR improves scores across the entire distribution. On FewRel-LT, it achieves an improvement of approximately 5% in Macro F1 score.
  • Efficiency: Significant gains are achieved by simply optimizing the classifier architecture, avoiding the high cost of retraining large language models with complex sampling schemes.
  • Robustness: The iterative routing process in ARR provides a more robust decision boundary, effectively mitigating the bias inherent in long-tailed data.

Conclusion

Our study shifts the focus from representation learning to classifier design in long-tailed relation classification. By decoupling these two stages, we identified the classifier as the primary source of bias. The proposed ARR mechanism offers a cost-effective and high-performance solution for extracting relations from imbalanced data.

Summary updated

Abstract

Long-tailed relation classification is a challenging problem as the head classes may dominate the training phase, thereby leading to the deterioration of the tail performance. Existing solutions usually address this issue via class-balancing strategies, e.g., data re-sampling and loss re-weighting, but all these methods adhere to the schema of entangling learning of the representation and classifier. In this study, we conduct an in-depth empirical investigation into the long-tailed problem and found that pre-trained models with instance-balanced sampling already capture the well-learned representations for all classes; moreover, it is possible to achieve better long-tailed classification ability at low cost by only adjusting the classifier. Inspired by this observation, we propose a robust classifier with attentive relation routing, which assigns soft weights by automatically aggregating the relations. Extensive experiments on two datasets demonstrate the effectiveness of our proposed approach. Code and datasets are available in https://github.com/zjunlp/deepke.
BibTeX
@misc{yu2020devil,
  author = {Haiyang Yu and Ningyu Zhang and Shumin Deng and Zonggang Yuan and Yantao Jia and Huajun Chen},
  title = {{The Devil is the Classifier: Investigating Long Tail Relation Classification with Decoupling Analysis}},
  year = {2020},
  eprint = {2009.07022},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.07022},
}
Figure