Expert-Agnostic Learning to Defer

Joshua Strong, Pramit Saha, Yasin Ibrahim, Cheng Ouyang, J. Alison Noble

Abstract

Learning to Defer (L2D) learns autonomous systems to independently manage straightforward cases, while deferring uncertain cases to human experts. Recent advancements in this field have introduced features enabling flexibility to unseen experts at test-time, but we find these approaches have significant limitations. To address these, we introduce EA-L2D: Expert-Agnostic Learning to Defer, a novel L2D framework that leverages a Bayesian approach to model expert behaviour in an expert-agnostic manner, facilitating optimal deferral decisions. EA-L2D offers several critical improvements over prior methods, including the ability to incorporate prior knowledge about experts, a reduced reliance on expert-annotated data, and robust performance when deferring to experts with expertise not seen during training. Evaluating on CIFAR-10, HAM10000, German Traffic Lights, Breast Ultrasound, Axial Organ Slices, and Blood Cell MNIST, we observe performance gains over the next state-of-the-art of 1-16% for seen experts and 4-28% for unseen experts in settings with high expert diversity.

Introduction

In high-stakes fields like medicine and autonomous driving, AI systems are often designed to work alongside human specialists. Learning to Defer (L2D) is a framework where a "rejector" model decides whether the AI should make a prediction or hand the decision over to a human expert.

A major hurdle for current L2D systems is the "unseen expert" problem. In real-world environments, such as a hospital, staff rotate frequently. A system trained on one group of doctors may struggle when a new specialist arrives with a different competence profile. This failure is often caused by identity leakage, where models learn brittle shortcuts based on fixed class indices (e.g., "Expert A is good at Class 3"). If the classes are reindexed or the expert's specialty shifts, these shortcuts break.

The EA-L2D Framework

To address these limitations, we introduce Expert-Agnostic Learning to Defer (EA-L2D). This framework utilizes an Identity-Free Deferral (IFD) architecture that enforces permutation symmetry—the idea that renaming or reordering classes should not change the deferral logic. By design, EA-L2D focuses on structural relationships rather than specific class identities.

1. Bayesian Competence Profiles

EA-L2D builds a profile of an expert’s performance using a few-shot "context" of their past decisions. By employing Beta-Binomial conjugacy, the framework provides a robust estimate of an expert's skill per class. This Bayesian approach captures both the mean performance and the uncertainty (variance) of the estimate, allowing the system to incorporate prior knowledge and handle sparse data effectively.

2. Role-Indexed State Representation

The core innovation of the IFD architecture is the transition from absolute class identities to "roles." Instead of feeding the rejector raw class IDs, the state is represented by structural values, such as the model’s confidence in its top-ranked class and the expert’s estimated skill for that specific class. This 6-dimensional state forces the model to learn universal rules—such as deferring when the expert's estimated competence significantly exceeds the model's confidence—regardless of the specific class label.

3. Uncertainty-Aware Training

The framework is trained using a context-only surrogate objective. By utilizing Lower Confidence Bounds (LCB) to weight deferral supervision, the system naturally downweights experts with uncertain profiles. This method significantly reduces the reliance on expensive expert-annotated data during query time, as the supervision is derived from the context-based profiles.

Experimental Evaluation

We evaluated EA-L2D across a diverse range of high-stakes benchmarks, including:

  • Medical Imaging: HAM10000 (dermatology), Breast Ultrasound, Axial Organ Slices, and Blood Cell MNIST.
  • General Benchmarking: CIFAR-10, German Traffic Lights, and ImageNet-16H (featuring real human annotations).

To rigorously test the system, we developed a new "instance-dependent" simulation protocol. This protocol models experts with varying specialties and difficulty-sensitive performance, providing a more realistic stress test than traditional uniform simulations.

Key Results and Insights

  • Superior Generalization to Unseen Experts: EA-L2D consistently outperformed state-of-the-art baselines (such as L2D-Pop). The most significant gains were observed in Out-of-Distribution (OOD) settings where expert specialties shifted, showing improvements of 4-28% for unseen experts.
  • Permutation Invariance: We formally demonstrate that standard population encoders are generally non-invariant to class relabeling. By design, EA-L2D is provably invariant, which explains its robust performance when encountering new experts or reindexed classes.
  • Robustness to Input Noise: On the ImageNet-16H dataset, EA-L2D maintained stable performance even as image corruption increased. Unlike baselines that re-embed noisy inputs for every decision, EA-L2D’s use of query-independent profiles provides a layer of protection against input degradation.
  • Data Efficiency: EA-L2D is significantly more annotation-efficient. In the Blood Cell MNIST dataset, it achieved superior results while requiring orders of magnitude fewer expert labels than competing methods.
  • Scaling with Context: Unlike previous meta-learning approaches that plateaued, EA-L2D’s performance improved monotonically as more context (past expert predictions) became available, effectively "learning" the expert's strengths in real-time.

Conclusion and Future Work

EA-L2D represents a shift toward more flexible and robust human-AI collaboration. By stripping away class identities and focusing on structural competence, the framework allows AI systems to effectively hand off decisions to experts they have never encountered before.

While EA-L2D currently operates at a class-level granularity, future work could explore "within-class" nuances—identifying specific instances where an expert might excel due to factors beyond the class label. Additionally, integrating instance-level features without re-introducing identity leakage remains a promising avenue for further research.

Summary updated

Abstract

Learning to Defer (L2D) learns autonomous systems to independently manage straightforward cases, while deferring uncertain cases to human experts. Recent advancements in this field have introduced features enabling flexibility to unseen experts at test-time, but we find these approaches have significant limitations. To address these, we introduce EA-L2D: Expert-Agnostic Learning to Defer, a novel L2D framework that leverages a Bayesian approach to model expert behaviour in an expert-agnostic manner, facilitating optimal deferral decisions. EA-L2D offers several critical improvements over prior methods, including the ability to incorporate prior knowledge about experts, a reduced reliance on expert-annotated data, and robust performance when deferring to experts with expertise not seen during training. Evaluating on CIFAR-10, HAM10000, German Traffic Lights, Breast Ultrasound, Axial Organ Slices, and Blood Cell MNIST, we observe performance gains over the next state-of-the-art of 1-16\% for seen experts and 4-28\% for unseen experts in settings with high expert diversity.
BibTeX
@inproceedings{strong2026expert,
  author = {Joshua Strong and Pramit Saha and Yasin Ibrahim and Cheng Ouyang and J. Alison Noble},
  title = {{Expert-Agnostic Learning to Defer}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=4YG9ufFg58},
}
Figure