One-Stage Top-k Learning-to-Defer: Score-Based Surrogates with Theoretical Guarantees

Axel CarlierarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces a novel "one-stage Top-k Learning-to-Defer" (L2D) framework that allows a model to simultaneously predict an outcome and decide which of k experts or labels to consult, all within a single training process. This unified approach improves upon existing methods by enabling more flexible and efficient decision-making in high-stakes scenarios, backed by theoretical guarantees.

What Problem Does This Paper Address?

In high-stakes decision-making (like medical diagnosis or financial forecasting), it's crucial to make accurate predictions while managing costs. Traditional "Learning-to-Defer" (L2D) methods allow a model to either make a prediction itself or defer to a single expert. However, this single-expert approach limits the ability to leverage the diverse strengths of multiple experts or labels. Existing "Top-k" L2D methods address this by allowing consultation of k experts, but they often use a two-stage process (training the predictor and then the deferral policy separately), which can lead to suboptimal performance. This paper tackles the challenge of creating a one-stage Top-k L2D framework that jointly optimizes prediction and deferral across multiple entities, offering a more integrated and efficient solution.

Methods

The core innovation is a one-stage Top-k Learning-to-Defer framework. Unlike previous methods that might train a predictor and then a deferral policy, this approach trains a single score-based model that simultaneously learns to:

  1. Predict class labels: Assigning a score to each potential class.
  2. Select entities (labels or experts): Assigning scores to k most promising entities (labels or experts) for consultation.

The authors introduce a cost-sensitive formulation where both labels and experts have associated costs (prediction error and consultation cost). They derive a novel convex surrogate loss that is independent of the cardinality k. This is a key advantage, as it allows the trained model to be used for different k values without retraining. They also propose an adaptive variant, Top-k(x), which dynamically selects the number of entities to consult per input based on its complexity, further optimizing the accuracy-cost trade-off. Theoretical guarantees of Bayes- and H-consistency are provided for their approach.

Datasets

The paper evaluates its methods on two standard image classification benchmarks:

  • CIFAR-10: A widely used dataset with 60,000 color images across 10 classes.
  • SVHN (Street View House Numbers): A large-scale digit classification benchmark with over 600,000 RGB images.

For experiments, the authors synthetically construct pools of experts with overlapping areas of competence on these datasets.

Key Insights and Ideas

  • Unified Prediction and Deferral: The one-stage approach is crucial. Jointly optimizing prediction and deferral allows the model to learn more nuanced strategies than separate training stages.
  • Score-Based Generalization to Top-k: Extending score-based L2D to Top-k is natural and powerful, enabling richer decision-making by considering multiple entities.
  • Cardinality-Independent Surrogate: A surrogate loss that doesn't depend on k is a significant practical advantage, allowing for flexible deployment across different Top-k scenarios without retraining.
  • Adaptive Consultation (Top-k(x)): Dynamically adjusting the number of consulted entities based on input complexity is key to achieving superior accuracy-cost trade-offs. Simple inputs might only need one consultation, while complex ones benefit from more.
  • Theoretical Guarantees: Providing Bayes- and H-consistency ensures that the learned model converges to the optimal deferral policy as more data becomes available.

Key Results

  • The proposed Top-k L2D framework strictly outperforms the Top-1 L2D baseline on CIFAR-10 and SVHN across various metrics (Top-k Accuracy, Majority Voting Accuracy, Weighted Voting Accuracy).
  • The adaptive Top-k(x) variant consistently dominates fixed-k strategies for a given budget level. For example, on CIFAR-10, Top-k(x) achieved a Majority Voting Accuracy of 95.53% at a budget of 0.192, outperforming the best fixed-k Top-k result of 94.7% which required a higher budget (0.199).
  • This performance gain is attributed to Top-k(x)'s ability to dynamically select the number of entities to consult, avoiding over-querying for simple inputs and ensuring sufficient consultation for complex ones.

Limitations and Future Work

  • Synthetic Experts: The study relies on synthetically constructed experts. While this allows for controlled experiments, evaluating on datasets with real human expert annotations would be valuable.
  • Uniform Cost Coefficients: The cardinality loss uses simplified uniform cost coefficients. Future work could explore more expressive cost structures.
  • Fixed Expert Behavior: The framework assumes fixed expert behavior during training and testing, and lacks support for online interaction with experts.
  • Future Directions: Extending Top-k and Top-k(x) L2D to settings with dynamic, queryable experts is suggested as an important avenue for future research.

Summary updated

BibTeX
@misc{carlier2025one,
  author = {Axel Carlier},
  title = {{One-Stage Top-k Learning-to-Defer: Score-Based Surrogates with Theoretical Guarantees}},
  year = {2025},
  eprint = {2505.10160},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.10160},
}
Figure