TL;DR
This paper critically evaluates existing metrics for "Ordinal Classification" (OC) and "Ordinal Quantification" (OQ) tasks, which deal with ordered categories (e.g., sentiment scales). The authors propose a set of recommendations for choosing the most appropriate metrics, highlighting that many commonly used metrics fail to account for the ordinal nature of the data, leading to potentially misleading evaluations.
What Problem Does This Paper Address?
In many machine learning tasks, especially in Natural Language Processing (NLP), we deal with data that has an inherent order. For example, sentiment analysis might classify text into categories like "highly negative," "negative," "neutral," "positive," and "highly positive." This is called Ordinal Classification (OC). A related task, Ordinal Quantification (OQ), involves estimating the distribution of data across these ordered categories.
The problem is that standard evaluation metrics, designed for nominal (unordered) categories, often fail to capture the nuances of ordinal data. For instance, misclassifying "highly positive" as "positive" is a much smaller error than misclassifying it as "highly negative." However, simple accuracy or other nominal metrics treat all misclassifications equally. This paper aims to systematically evaluate existing metrics for OC and OQ tasks to understand their properties and provide guidance on selecting appropriate ones that truly reflect performance on ordinal data.
Methods
The authors conducted a meta-evaluation of various metrics using data from established NLP challenges like SemEval and NTCIR. They analyzed nine metrics for OC tasks and six for OQ tasks. The evaluation focused on several key properties:
- Ordinality Handling: Do the metrics penalize misclassifications based on the distance between ordinal classes?
- Baseline System Handling: Can the metrics correctly evaluate systems that always predict the same class?
- Class Imbalance Handling: Do the metrics account for uneven distributions of classes in the data?
- System Ranking Similarity: How similar are the rankings of systems produced by different metrics?
- System Ranking Consistency (Robustness): How stable are the rankings produced by a metric across different subsets of the data?
- Discriminative Power: How well can a metric distinguish between the performance of different systems?
They employed statistical tests, such as Kendall's tau and Tukey's HSD, to rigorously compare the behavior of these metrics.
Datasets
The paper utilizes publicly available datasets from several well-known NLP evaluation campaigns:
- SemEval-2016/2017 Task 4 (Sentiment Analysis in Twitter): Subtask C for Ordinal Classification (5 classes) and Subtask E for Ordinal Quantification (5 classes).
- NTCIR-15 DialEval-1 (Dialogue Quality): Subtasks A, E, and S for Ordinal Quantification (5 classes each). These are larger datasets and were prioritized for OQ evaluation.
The authors note that they did not use Arabic data from SemEval 2017 due to a low number of submissions.
Key Insights and Ideas
- Many common metrics are "axiomatically incorrect" for ordinal tasks: Metrics like standard Accuracy and Mean Absolute Error (MAE) treat all classes as nominal, ignoring the inherent order. This can lead to misleading conclusions about system performance.
- The "distance" between classes matters: Metrics that incorporate the ordinal nature (e.g., penalizing larger gaps more) are crucial for accurate evaluation.
- Baseline systems pose a challenge: Some metrics (like Weighted Kappa) struggle to differentiate between systems that consistently predict the same class, which can be unrealistic but sometimes present in benchmark datasets.
- Symmetry is not always necessary for OQ: While some OQ metrics are symmetric, the authors found that symmetry doesn't necessarily improve performance and can even be detrimental.
- Consistency and discriminative power are key: Metrics that produce stable rankings across data subsets and can effectively distinguish between system performances are preferred.
Key Results
The paper provides extensive comparative results across various metrics and datasets. Some highlights include:
- For OC tasks:
- Metrics like Weighted Kappa (Linear Weighted) and Krippendorff's alpha (a-ORD) were recommended as primary measures, with a-ORD being particularly useful when baseline systems are present.
- Metrics that ignore ordinality (MAE, Accuracy) showed significantly different rankings and were deemed less useful.
- CEMORD, while designed for OC, showed less consistent ranking similarity and discriminative power compared to other ordinal-aware metrics in their experiments.
- For OQ tasks:
- RNOD (Root Normalised Order-aware Divergence) was recommended as the primary measure, with NMD (Normalised Match Distance) as a secondary choice.
- NMD was found to be an outlier in terms of system ranking similarity, often producing different results than other axiomatically correct OQ measures.
- Introducing symmetry to RNOD (resulting in RSNOD) did not appear to be beneficial.
Limitations and Future Work
- Limitations:
- The authors acknowledge that their meta-evaluation relies on specific datasets and that results might vary with different data distributions or task types.
- The study focuses on specific properties of metrics; other aspects like computational efficiency are not discussed.
- The "equidistance" assumption in some recommended metrics (like Linear Weighted Kappa and RNOD) might not always hold true for all ordinal tasks, though the authors suggest this can be addressed by adjusting weights.
- Future Work:
- The authors plan to extend their evaluation to tasks beyond OC and OQ.
- They suggest that researchers should always understand the properties of the evaluation measures they use to ensure they are appropriate for their specific task.
- The authors have made their evaluation measure score matrices publicly available to facilitate reproducibility.
Summary updated