Coherent Hierarchical Multi-Label Learning to Defer for Medical Imaging

Joshua Strong, Emma Sun, Pramit Saha, Helen Higham, J. Alison Noble

Abstract

Learning to Defer (L2D) enables a model to predict autonomously or defer to an expert, but prior work largely assumes flat label spaces. We study the first L2D setting with hierarchical multi-label decisions, motivated by medical-imaging workflows in which findings are organised by clinical taxonomies. In this setting, deferral is a delegation action rather than a label assignment, so treating it as an independent per-label decision can produce deferral incoherence, including taxonomic contradictions, delegation violations, and deferrals of labels already implied by the model's own assertions. We formalise coherent hierarchical deferral under a Selective-Exclusion handoff contract, characterise the Bayes-optimal coherent deferral rule, and show that even nodewise Bayes L2D can be action-incoherent. We then propose two remedies: exact coherent projection, a dynamic-programming decoder over the coherent action set, and Taxonomic Belief Propagation (TBP) with Recursive Policy Optimisation (RPO), a contract-aware joint action model trained through the same recursion used at inference. Across real-reader and controlled-expert medical-imaging benchmarks, naive binary-relevance L2D exhibits non-trivial incoherence. Projection removes it exactly, and fast TBP+RPO drives incoherence near zero while retaining strong utility.
BibTeX
@misc{strong2026coherent,
  author = {Joshua Strong and Emma Sun and Pramit Saha and Helen Higham and J. Alison Noble},
  title = {{Coherent Hierarchical Multi-Label Learning to Defer for Medical Imaging}},
  year = {2026},
  eprint = {2605.02734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.02734},
}
Figure