Abstract
Large-Scale Multi-Label Text Classification (LMTC) includes tasks with hierarchical label spaces, such as automatic assignment of ICD-9 codes to discharge summaries. Performance of models in prior art is evaluated with standard precision, recall, and F1 measures without regard for the rich hierarchical structure. In this work we argue for hierarchical evaluation of the predictions of neural LMTC models. With the example of the ICD-9 ontology we describe a structural issue in the representation of the structured label space in prior art, and propose an alternative representation based on the depth of the ontology. We propose a set of metrics for hierarchical evaluation using the depth-based representation. We compare the evaluation scores from the proposed metrics with previously used metrics on prior art LMTC models for ICD-9 coding in MIMIC-III. We also propose further avenues of research involving the proposed ontological representation.
Introduction
In Large-Scale Multi-Label Text Classification (LMTC), models must often select from thousands of labels organized in complex hierarchies. A prominent example is ICD-9 coding, where clinical notes are tagged with standardized medical codes. Traditionally, these models are evaluated using "flat" metrics like Precision, Recall, and F1. These metrics treat all errors equally: misidentifying a specific heart condition as a closely related one is penalized as heavily as misidentifying it as an unrelated respiratory issue.
While existing hierarchical metrics (set-based) attempt to address this by granting partial credit for correctly identifying ancestor nodes, they remain binary at every level. They only track if at least one descendant was predicted, failing to capture whether a model over-predicts or under-predicts within a specific medical family. This leads to a loss of information regarding model calibration and reliability in high-stakes clinical settings.
Methods
The authors introduce CoPHE (Count-Preserving Hierarchical Evaluation), which shifts from binary set-based logic to a count-preserving approach. The methodology consists of three main components:
- Level-Based Augmentation: To handle the inconsistent depths of the ICD-9 ontology, labels are organized into distinct levels based on code structure (e.g., category vs. etiology).
- Count-Preserving Logic: Unlike set-based metrics where a parent node is "True" if any child is predicted, CoPHE assigns the parent node a value equal to the sum of its predicted descendants.
- Redefined TP/FP/FN: Precision and Recall are calculated using
minandmaxfunctions to determine True Positives (overlap between predicted and actual counts), False Positives (over-predictions), and False Negatives (under-predictions).
This approach allows the metric to penalize "hallucinations"—instances where a model predicts multiple codes within a family when only one or two are present—even if the parent category is technically correct.
Experimental Results
The researchers evaluated CoPHE using the MIMIC-III dataset, focusing on the ICD-9 ontology across the top-50 most frequent codes and the full long-tailed distribution. They compared three state-of-the-art models: CAML, BGLWAN, and HLAN.
Key findings include:
- Metric Leniency: Hierarchical measures (Set-Based and CoPHE) consistently report higher F1 scores than flat measures. However, CoPHE scores are consistently lower than Set-Based scores, confirming that binary set-based metrics overestimate performance by ignoring over-predictions.
- Model Ranking Stability: While the absolute scores changed, the relative ranking of the models remained stable across metrics. This suggests that current architectures are not yet effectively exploiting hierarchical structures during training.
- Granular Error Tracking: By preserving counts, CoPHE provides a more nuanced view of model behavior, distinguishing between missing a category entirely and simply miscounting the specific conditions within it.
Conclusion
CoPHE provides a middle ground between the strictness of flat metrics and the leniency of set-based hierarchical metrics. By preserving counts, it offers a more realistic assessment of model performance in domains like medical billing and clinical decision support. Future work involves extending CoPHE to non-tree graph structures like SNOMED CT and integrating count-preserving logic directly into model loss functions to encourage better hierarchical learning.
Summary updated