TL;DR
This paper argues that relying solely on micro-F1 for evaluating relation classification models on imbalanced datasets is insufficient. They introduce a framework for class weighting schemes, including two novel methods (dodrans and entropy), to provide a more comprehensive view of model performance, especially for rare relations.
What Problem Does This Paper Address?
Relation classification (RC) models are crucial for extracting structured information from text, identifying relationships between entities. However, a common practice in evaluating these models is to report a single metric, often micro-F1. This is problematic because many real-world RC datasets are highly imbalanced, meaning some relation types appear far more frequently than others. Micro-F1, which aggregates performance across all instances, can mask poor performance on rare, "long-tail" relations, which are often the most interesting and challenging to detect. This can lead to favoring models that are good at predicting common relations but fail on less frequent ones, hindering progress in real-world applications where detecting these rare phenomena is critical.
Methods
The authors propose a framework for class weighting schemes to address the limitations of single-metric evaluation. This framework is built upon desiderata for weighting functions, including normalization (sum of weights equals 1), monotonicity (higher class count implies higher or equal weight), and a specific ratio between weights and counts.
They introduce two novel weighting schemes:
- Dodrans: This scheme assigns weights proportional to $n^{3/4}$, where $n$ is the number of instances in a class. This is inspired by theoretical work on optimizing for imbalanced datasets.
- Entropy: This scheme weights classes based on their contribution to the Shannon entropy of the class distribution, aiming to capture the difficulty of predicting a class.
These new schemes, along with existing micro- and macro-weighting, are used to calculate F1-scores. Micro-F1 aggregates performance across all instances, effectively treating each instance equally. Macro-F1 gives equal weight to each class, regardless of its size, highlighting performance on rare classes. The proposed dodrans and entropy schemes fall in between these extremes, offering a more nuanced evaluation.
Datasets
The paper evaluates their proposed methods on two widely used RC datasets:
- TACRED: A large-scale dataset with a significant number of relation types and a long-tail distribution.
- SemEval-2010 Task 8: Another benchmark dataset for relation classification.
Key Insights and Ideas
- Micro-F1 is a poor indicator for imbalanced datasets: Relying solely on micro-F1 can be misleading, as it can hide poor performance on rare relations.
- Multiple weighting schemes reveal different model strengths: Different weighting schemes highlight different aspects of a model's performance. For instance, macro-F1 is sensitive to performance on rare classes, while micro-F1 reflects overall performance.
- Dodrans and Entropy offer a balanced perspective: The proposed dodrans and entropy weighting schemes provide a middle ground, offering a more comprehensive view of model performance than just micro or macro.
- Long-tail performance is crucial: The paper emphasizes the importance of evaluating performance on rare relations, as these are often critical for real-world applications.
- Prompt-tuning models show better generalization: Models using prompt-tuning (like PTR) tend to have more stable performance across different weighting schemes, suggesting better generalization, especially for rare classes.
Key Results
- TACRED: PTR significantly outperforms RECENT across all weighting schemes. The performance gap widens as weighting schemes become more class-equal (e.g., macro-F1 shows the largest difference).
- SemEval: BERTEM outperforms PTR on micro-F1 and most other weightings, but PTR shows a significant advantage on macro-F1, indicating BERTEM struggles more with rare relations.
- Dodrans and Entropy are informative: The proposed dodrans and entropy schemes consistently showed similar results to each other and provided insights that were not captured by micro or macro alone. For example, on TACRED, the difference between PTR and RECENT was smallest for micro-F1 and largest for macro-F1, with dodrans and entropy falling in between.
Limitations and Future Work
- Coincidence of datasets: The authors acknowledge that the similar results of dodrans and entropy might be a coincidence of the chosen datasets and suggest further investigation.
- Optimization for specific weightings: They note that weighting schemes can be optimized for during training, which could further bias results. This reinforces their argument for reporting multiple measures.
- Extension to other NLP tasks: The analysis and proposed framework can be extended to other NLP tasks with imbalanced datasets, such as named entity recognition, part-of-speech tagging, and coreference resolution.
- Bidimensional leaderboard: The authors suggest creating a bidimensional leaderboard that considers multiple evaluation measures to provide a more holistic view of model performance.
- Reproducibility: They highlight the reproducibility crisis and suggest that future work should focus on comprehensive evaluation studies that include reproducible results.
Summary updated