Introduction
Learning to Defer (L2D) is a collaborative framework where an AI system and a human expert work together to make decisions. In high-stakes domains such as medical diagnostics or autonomous navigation, the system must decide whether to provide its own prediction or "defer" the task to a human expert. A significant challenge in training effective L2D models is the requirement for large datasets of expert predictions to model the expert's strengths and weaknesses. Collecting thousands of such labels is often prohibitively expensive or time-consuming. This paper addresses this "data hunger" by proposing a method to learn an expert's capabilities from a minimal sample of their work.
Methodology: A Three-Step Framework
The authors introduce a pipeline designed to "hallucinate" expert behavior across an entire dataset using only a handful of actual expert labels. This process is divided into three distinct phases:
- Feature Extraction (Embedding Model): A standard classification model is trained on ground truth labels to learn a rich feature representation (embeddings) of the data.
- Expertise Prediction (The "Expert Mimic"): A secondary model is trained to predict the binary outcome of whether a human expert will be correct or incorrect for a given input. To maximize efficiency, this step utilizes Semi-Supervised Learning (SSL) techniques like FixMatch or CoMatch, leveraging patterns in unlabeled data to generalize from a few expert examples.
- Artificial Label Generation: The "Expert Mimic" generates predicted human labels for the full training set. If the mimic predicts the expert would be correct, it assigns the ground truth label; otherwise, it assigns a random incorrect label.
This generated dataset allows any standard L2D algorithm to be trained as if a complete set of expert predictions were available.
Experimental Evaluation
The framework was evaluated on two datasets:
- CIFAR-100: Used to simulate synthetic experts with specific categorical biases (e.g., high accuracy on animals, low accuracy on vehicles).
- NIH Chest X-ray Dataset: A real-world medical dataset featuring predictions from individual radiologists compared against "gold standard" panel labels.
Key Results and Insights
The proposed "Embedding-SSL" approach demonstrates remarkable data efficiency. On the NIH dataset, the system achieved 98.83% of the performance of a model trained on the full expert dataset while using only six expert predictions per class.
Key insights from the study include:
- Binary Simplification: Modeling expertise as a binary "correct/incorrect" task significantly simplifies the learning problem compared to predicting specific class errors.
- Transferability of Features: Features learned for object recognition are highly predictive of human error, particularly near decision boundaries where both AI and humans tend to struggle.
- SSL as a Catalyst: Semi-supervised learning allows the model to map expert behavior across the entire latent space, rather than just memorizing specific points.
Conclusion and Future Directions
This research provides a viable path for deploying AI-human collaborative systems in data-scarce environments. By reducing the expert labeling requirement by orders of magnitude, L2D becomes practical for specialized medical tasks or rapidly changing expert pools. Future work may explore Active Learning to select the most informative samples for expert labeling and extending the framework to handle multiple experts simultaneously.
Summary updated