Abstract
Learning to Defer (L2D) improves AI reliability in decision-critical environments by training AI to either make its own prediction or defer the decision to a human expert. A key challenge is adapting to unseen experts at test time, whose competence can differ from the training population. Current methods for this task, however, can falter when unseen experts are out-of-distribution (OOD) relative to the training population. We identify a core architectural flaw as the cause: they learn identity-conditioned policies by processing class-indexed signals in fixed coordinates, creating shortcuts that violate the problem's inherent permutation symmetry. We introduce Identity-Free Deferral (IFD), an architecture that enforces this symmetry by construction. From a few-shot context, IFD builds a query-independent Bayesian competence profile for each expert. It then supplies the deferral rejector with a low-dimensional, role-indexed state containing only structural information, such as the model's confidence in its top-ranked class and the expert's estimated skill for that same role, which obscures absolute class identities. We train IFD using an uncertainty-aware, context-only objective that removes the need for expensive query-time expert labels. We formally prove the permutation invariance of our approach, contrasting it with the generic non-invariance of standard population encoders. Experiments on medical imaging benchmarks and ImageNet-16H with real human annotators show that IFD consistently improves generalisation to unseen experts, with gains in OOD settings, all while using fewer annotations than alternative methods.
Introduction
In high-stakes fields like healthcare, AI shouldn't always make the final call. Learning to Defer (L2D) trains an AI to handle the easy cases autonomously while routing the hard ones to a human expert.
However, real-world environments are dynamic. A hospital might hire a new doctor whose specific sub-specialty the AI has never seen before. The current state-of-the-art method for handling populations of experts (L2D-Pop) uses meta-learning to create latent "embeddings" of expert behavior. This approach often suffers from poor out-of-distribution (OOD) generalization, high data requirements, and an inability to incorporate prior knowledge about an expert's specialty.
The EA-L2D Framework
To solve these issues, we propose Expert-Agnostic Learning to Defer (EA-L2D), also known as Identity-Free Deferral. This framework abandons black-box meta-learning in favor of an explicit, interpretable Bayesian approach.
- Constructed Bayesian Representations: Instead of learning a latent vector, EA-L2D uses a small "context" set of an expert's past predictions. It uses a Beta-Binomial statistical model to explicitly calculate the expert's expected accuracy (mean) and uncertainty (variance) for every single class.
- Incorporating Priors: Because it uses a Bayesian framework, EA-L2D can easily incorporate prior knowledge. If an expert self-reports high confidence in a specific area, the system uses that as a mathematical prior, which is incredibly useful when historical data is sparse.
- Expert-Agnostic Rejector: The AI uses a "rejector" network to decide whether to defer. Crucially, this network is blinded to the actual class labels and expert IDs. It only sees the structural math: “My confidence is X, and the expert’s proven accuracy for my predicted class is Y ± Z.” This forces the AI to learn universal rules of collaboration rather than memorizing specific expert profiles.
- Uncertainty-Aware Loss: The training loss function uses a Lower Confidence Bound (LCB). If the AI only has limited data for an expert, the LCB penalizes this uncertainty, preventing the AI from blindly trusting a "lucky" but low-skill expert.
Experimental Setup
The method was evaluated on four diverse medical imaging benchmarks: HAM10000 (skin lesions), Blood Cells (microscopic images), Retinal OCT (eye scans), and Liver Tumours (radiological CT volumes). Additionally, the BCN20000 dataset was used to test how well the models handle distribution shifts compared to training data.
Key Results
EA-L2D consistently outperforms existing baselines, particularly when dealing with experts whose skills differ from those seen during training.
- Superior OOD Performance: EA-L2D achieved up to a 28% relative improvement in deferral accuracy (AURDAC) when tested on unseen, out-of-distribution experts compared to L2D-Pop.
- Massive Data Efficiency: EA-L2D completely removes the need for expert annotations on the large "query" training set. It only needs a tiny "context" set of labels per expert, drastically reducing the cost of training.
- Robustness to Input Shifts: When tested on the entirely different BCN20000 dataset, EA-L2D maintained significantly higher system and deferral accuracy than baselines, proving its resilience to real-world domain shifts.
Conclusion
By forcing the model to represent experts via explicit statistics rather than abstract latent vectors, EA-L2D avoids entangling an expert's skill level with specific class identities. This agnosticism ensures the learned deferral logic is universally transferable. While inaccurate priors can initially impact performance, the Bayesian model naturally adapts as it observes more real-world data, making it a robust choice for dynamic clinical environments.
Summary updated