CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification

Matús Falis, Hang Dong, Alexandra Birch, Beatrice Alex

Abstract

Large-Scale Multi-Label Text Classification (LMTC) includes tasks with hierarchical label spaces, such as automatic assignment of ICD-9 codes to discharge summaries. Performance of models in prior art is evaluated with standard precision, recall, and F1 measures without regard for the rich hierarchical structure. In this work we argue for hierarchical evaluation of the predictions of neural LMTC models. With the example of the ICD-9 ontology we describe a structural issue in the representation of the structured label space in prior art, and propose an alternative representation based on the depth of the ontology. We propose a set of metrics for hierarchical evaluation using the depth-based representation. We compare the evaluation scores from the proposed metrics with previously used metrics on prior art LMTC models for ICD-9 coding in MIMIC-III. We also propose further avenues of research involving the proposed ontological representation.

Introduction

In Large-Scale Multi-Label Text Classification (LMTC), such as the automated assignment of ICD-9 medical codes to hospital discharge summaries, the label space is inherently structured. For instance, a code for Endomyocardial fibrosis is semantically and structurally closer to Endocardial fibroelastosis than to an unrelated condition like Tobacco use disorder.

Despite this, most existing research evaluates models using "flat" metrics—standard Precision, Recall, and F1 scores. These metrics treat a "near-miss" (predicting a closely related sibling) the same as a complete miss. While some hierarchical metrics exist, they often rely on binary logic that fails to account for the number of specific conditions predicted within a category, leading to a loss of information regarding model over-prediction or under-prediction.

Methodology

To address these limitations, this paper introduces two primary contributions: Level-Based Augmentation and the Count-Preserving Hierarchical Evaluation (CoPHE) metric.

Level-Based Augmentation

Standard ontologies like ICD-9 often contain structural inconsistencies where leaf nodes exist at varying depths. To resolve this, the authors redefine the hierarchy into strict depth levels (e.g., chapter, category, single-digit etiology). This ensures that when predictions are aggregated up the tree, they align consistently at the same depth.

Count-Preserving Hierarchical Evaluation (CoPHE)

Unlike traditional set-based hierarchical metrics that use binary flags for ancestor nodes, CoPHE preserves the count of descendants.

  • True Positives (TP) for an ancestor are calculated as the minimum of the predicted and true descendant counts.
  • False Positives (FP) and False Negatives (FN) are derived from the differences between these counts.

This approach ensures that if a model predicts multiple specific variations of a disease when only one is present, it is correctly penalized for over-prediction at the ancestor level.

Comparison of evaluation logic for Leaf-Only (A), Set-Based (B), and CoPHE (C) metrics.

Results and Discussion

The authors re-evaluated three state-of-the-art LMTC models—CAML, BGLWAN, and HLAN—on the MIMIC-III top-50 codeset benchmark using Flat, Set-Based, and CoPHE metrics.

Performance comparison of SOTA models across different evaluation metrics.

Key findings include:

  • CoPHE vs. Flat Metrics: CoPHE F1 scores (63.1% – 66.5%) were consistently higher than flat F1 scores (60.1% – 64.2%). This demonstrates CoPHE's intended leniency for near-misses within the same code family, which is valuable for semi-automated clinical workflows.
  • CoPHE vs. Set-Based Metrics: CoPHE F1 scores were consistently lower than set-based F1 scores (64.2% – 68.5%). This confirms that CoPHE successfully captures and penalizes over-predictions that binary set-based metrics ignore.
  • Model Rankings: The relative ranking of the models remained stable across all metrics, suggesting that current models do not yet inherently exploit hierarchical structures more effectively than their peers.

Conclusion

CoPHE provides a more nuanced and realistic evaluation of model performance in hierarchical label spaces by accounting for ontology depth and preserving prediction counts. While current models show consistent rankings across metrics, CoPHE highlights the need for future architectures that are explicitly designed to optimize for hierarchical structures. Future work will involve extending CoPHE to non-tree graph ontologies like SNOMED CT and evaluating models that incorporate ontological structure during training.

Summary updated

Abstract

Large-Scale Multi-Label Text Classification (LMTC) includes tasks with hierarchical label spaces, such as automatic assignment of ICD-9 codes to discharge summaries. Performance of models in prior art is evaluated with standard precision, recall, and F1 measures without regard for the rich hierarchical structure. In this work we argue for hierarchical evaluation of the predictions of neural LMTC models. With the example of the ICD-9 ontology we describe a structural issue in the representation of the structured label space in prior art, and propose an alternative representation based on the depth of the ontology. We propose a set of metrics for hierarchical evaluation using the depth-based representation. We compare the evaluation scores from the proposed metrics with previously used metrics on prior art LMTC models for ICD-9 coding in MIMIC-III. We also propose further avenues of research involving the proposed ontological representation.
BibTeX
@inproceedings{falis2021cophe,
  author = {Matús Falis and Hang Dong and Alexandra Birch and Beatrice Alex},
  title = {{CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=HUR6uPxnYv},
}
Figure