Coherent Hierarchical Multi-Label Learning to Defer for Medical Imaging

Joshua Strong, Pramit Saha, Emma Sun, Helen Higham, Alison Noble

TL;DR

This paper introduces the first Learning to Defer (L2D) framework designed for hierarchical, multi-label clinical taxonomies in medical imaging. By recognizing that deferral is a structural delegation action rather than a simple label, the authors expose critical "deferral incoherences" in standard independent models and propose two novel, contract-aware solutions that guarantee logical and operationally sound human-AI collaboration.

What Problem Does This Paper Address?

Learning to Defer (L2D) is a framework where an AI model can choose to either make an autonomous prediction or hand the decision over to a human expert (like a radiologist) when its uncertainty is high. However, existing L2D methods assume a "flat" label space, treating every condition as an independent choice.

In clinical practice, medical findings are inherently hierarchical and multi-label. For example, a broad finding like Lung Opacity serves as a parent category for more specific subtypes like Edema, Infiltration, and Consolidation.

Figure 1: Illustrative subtree used throughout Section 3.

If an AI makes independent deferral decisions across this tree, it can produce highly confusing, contradictory handoffs for a clinician. For instance, the AI might autonomously declare that Lung Opacity is absent, yet defer the decision on Edema to the human. Because an absent parent logically rules out all descendants, deferring the child node is a waste of human effort and structurally incoherent. This paper formalizes these failures and establishes rules for coherent human-AI handoffs over clinical taxonomies.

Methods

The authors model hierarchical deferral under a Selective-Exclusion (SE) handoff contract: if the AI defers a parent concept, it is allowed to selectively exclude specific subtypes, but it cannot assert any subtype as present (since asserting a child would logically force the deferred parent to be present).

They identify three distinct types of local edge failures that must be avoided:

  1. Taxonomic Contradiction: Asserting a child is present while ruling out the parent ($a_p = 0 \land a_c = 1$).
  2. Delegation Violation: Asserting a child is present while deferring the parent ($a_p = \perp \land a_c = 1$).
  3. Deductive Defect: Deferring a child when the parent has already been ruled out ($a_p = 0 \land a_c = \perp$).

To enforce this contract, the paper proposes two solutions:

  • Exact Coherent Projection: An inference-only method. It takes the independent probabilities from a standard pre-trained model and uses dynamic programming to find the exact Maximum A Posteriori (MAP) action vector that satisfies the SE contract, guaranteeing zero incoherence at test time.
  • Taxonomic Belief Propagation (TBP) with Recursive Policy Optimisation (RPO): A joint action model that embeds the hierarchical constraints directly into the network architecture using a top-down Markov recursion. By training end-to-end with RPO, the model learns the downstream risk and "option value" of its high-level deferral choices.
Table 1: Complete local classification of parent–children handoff decisions under Selective- Exclusion. Within each parent block, the Status column reports the first failed coherence criterion: taxonomic satisfiability, contract admissibility, or deductive closure.

Datasets

The framework was validated across four medical imaging datasets representing different scales, experts, and modalities:

  • VinDr-CXR & CheXpert: Primary chest X-ray benchmarks utilizing real-world, retrospective radiologist annotations.
  • PadChest: An extra-large chest X-ray taxonomy used as a stress test for scalability.
  • ADPv2: A histopathology tissue taxonomy used to evaluate the framework outside of chest radiography.

Key Insights and Ideas

  • Oracle Models Can Still Fail: The authors prove theoretically that even a perfectly calibrated, nodewise Bayes-optimal model can produce delegation violations and deductive defects if it evaluates nodes independently. Incoherence is a structural flaw of flat L2D, not just a training artifact.
  • Downstream Option Value: Deferral decisions are non-separable. A model must account for the downstream consequences of its choices; deferring a parent node might look locally optimal due to high expert confidence, but committing to an autonomous parent prediction might unlock a highly accurate autonomous prediction for a specific subtype downstream.
  • Operational vs. Logical Failures: Deferral incoherence isn't just about bad label predictions—it's an operational failure of the human-AI workflow. Incoherent action vectors obscure exactly what the model is taking responsibility for, defeating the purpose of an assistive tool.

Key Results

  • Baseline Incoherence: Naive binary-relevance L2D models consistently exhibit high rates of incoherence across all datasets (averaging 5% to 13% of decisions containing defects).
  • Perfect Correction: Exact Coherent Projection successfully eliminates 100% of deferral incoherence across the sweep of deferral budgets without requiring any model retraining.
  • Superior Utility: The joint TBP+RPO model drives incoherence down to near-zero while achieving significantly higher system utility (higher F1 scores and balanced accuracy) than both the naive baselines and the projection method.
  • Massive Speedups: At deployment time, TBP+RPO's fast marginal decoder runs up to 1074× faster per study than Exact Coherent Projection, offering a highly practical, high-throughput operating point for real-time clinical systems.

Limitations and Future Work

  • Tree Taxonomies Only: The current formulation assumes the clinical taxonomy is structured as a tree (where each node has exactly one parent). While common, many realistic clinical vocabularies are structured as Directed Acyclic Graphs (DAGs) where a finding can have multiple parents. The authors outline a theoretical extension to DAGs in the appendix, but practical implementation remains future work.
  • Static Expert Modeling: The framework assumes a stable expert profile and does not explicitly account for real-time human-taxonomy disagreements or shifting expert skill levels during live clinical deployment.

Summary updated

Abstract

Learning to Defer (L2D) enables a model to predict autonomously or defer to an expert, but prior work largely assumes flat label spaces. We study the first L2D setting with hierarchical multi-label decisions, motivated by medical-imaging workflows in which findings are organised by clinical taxonomies. In this setting, deferral is a delegation action rather than a label assignment, so treating it as an independent per-label decision can produce deferral incoherence, including taxonomic contradictions, delegation violations, and deferrals of labels already implied by the model's own assertions. We formalise coherent hierarchical deferral under a Selective-Exclusion handoff contract, characterise the Bayes-optimal coherent deferral rule, and show that even nodewise Bayes L2D can be action-incoherent. We then propose two remedies: exact coherent projection, a dynamic-programming decoder over the coherent action set, and Taxonomic Belief Propagation (TBP) with Recursive Policy Optimisation (RPO), a contract-aware joint action model trained through the same recursion used at inference. Across real-reader and controlled-expert medical-imaging benchmarks, naive binary-relevance L2D exhibits non-trivial incoherence. Projection removes it exactly, and fast TBP+RPO drives incoherence near zero while retaining strong utility.
BibTeX
@misc{strong2026coherent,
  author = {Joshua Strong and Pramit Saha and Emma Sun and Helen Higham and Alison Noble},
  title = {{Coherent Hierarchical Multi-Label Learning to Defer for Medical Imaging}},
  year = {2026},
  eprint = {2605.02734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.02734},
}
Figure