TL;DR
RACER (Role-Aligned Competence Estimation for Routing) is a novel framework for population-adaptive "learning to defer" (L2D) that dynamically decides whether an AI or an unseen human expert should handle a given query. By estimating expert competence on a true probability scale relative to candidate class roles rather than absolute class identities, RACER achieves instance-dependent routing while remaining robustly invariant to arbitrary class relabellings.
What Problem Does This Paper Address?
In high-stakes fields like healthcare, AI models shouldn't always make the final call; instead, they should know when to defer to a human expert. Traditional Learning to Defer (L2D) systems assume a fixed, known expert during training. However, real-world deployments face rotating clinicians, newly hired specialists, or changing shifts. This requires population-adaptive L2D, where the system must infer the competence of an unseen expert at test time using only a small "context set" of their past decisions.
Prior approaches suffer from a strict dichotomy:
- Expressive but fragile methods (e.g., L2D-Pop): These use neural context encoders to route instance-by-instance, but they expose absolute class coordinates. This allows the network to learn brittle shortcuts tied to specific class indices or names rather than transferable expert skills.
- Robust but rigid methods (e.g., Identity-Free Deferral - IFD): These enforce invariance to class names by building classwise competence profiles. However, their estimates are constant within each class, failing to capture instance-level expert specialization (e.g., an expert who is great at typical cases of a disease but struggles with atypical ones).
RACER bridges this gap, achieving both query-dependent adaptation and identity-free transfer simultaneously.
Methods
RACER separates classification, expert-competence estimation, and routing into distinct steps. Instead of mapping an expert's context to an unconstrained latent embedding, RACER estimates a posterior-predictive competence functional $\Gamma(x, y, C_e) \in [0, 1]$. This functional represents the probability that an expert (characterized by context $C_e$) will be correct on a query $x$, assuming the true class role is $y$.
The framework introduces a same-role kernel pooling primitive:
- It computes the cosine similarity between the query $x$ and context examples.
- It aggregates similarity masses only for context examples where the expert's true label matches the candidate role $y$.
The authors propose two main instantiations:
- Racer-KNN: A transparent, nonparametric approach that takes a uniform average of the nearest same-role context neighbors.
- Racer-kernel (Default): A neural approach that feeds role-relative summaries (such as local kernel statistics, posterior margins, ranks, and query uncertainty) into a small, role-shared Multi-Layer Perceptron (MLP).
To ensure these estimates behave like true probabilities, the competence head is trained using a strictly proper scoring rule (binary cross-entropy) with stop-gradients blocking routing-loss gradients from corrupting the competence parameters.
Datasets
The authors evaluated RACER on both controlled synthetic environments and real-world clinical datasets:
- PathMNIST: A histopathology dataset used for a context-scaling study with simulated experts exhibiting hidden within-class subtype specializations.
- CIFAR-100: Used to stress-test class cardinality ($K=20$ and $K=100$) and expert-profile permutations.
- VinDr-CXR: A real-world, multi-radiologist chest X-ray dataset used to evaluate routing performance under realistic expert variability and annotation noise.
- CheXpert: A multi-expert human-AI chest radiography benchmark containing annotations from multiple human readers and external pretrained AI systems.
Key Insights and Ideas
- Role-Relative vs. Absolute Coordinates: By evaluating competence relative to a candidate class role (e.g., "how well does the expert perform when the true label is $y$?") rather than absolute class indices, RACER inherently satisfies coherent class-relabelling invariance. If you rename the classes, the routing decisions remain semantically identical.
- Probability-Scale Calibration: Unlike methods that output arbitrary latent routing scores, RACER targets the true conditional probability of expert correctness. This explicit calibration is crucial for real-world triage, multi-expert routing, and threshold selection under varying operational budgets.
- Resolving Within-Class Bias: Classwise profiles (like IFD) fail when an expert's skill varies within a class due to image quality, subtype variations, or local morphology. RACER's query-conditioned kernel pooling successfully extracts local context evidence to resolve this within-class bias.
Key Results
- Context Scaling Wins: In the PathMNIST study under hidden subtype dependence, Racer-kernel demonstrated a clear context-scaling trend. While classwise profiles plateaued due to within-class bias, Racer-kernel converted larger context sets into substantial routing utility gains, crossing the classifier baseline at $B/K \approx 5.6$.
- Robustness to Cardinality: On the CIFAR-100 benchmark, Racer-kernel was the only evaluated method to achieve a positive aggregate out-of-distribution (OOD) routing gain over the classifier baseline across both large and small label spaces ($K=20$ and $K=100$).
- Clinical Relevance: On the real-radiologist VinDr-CXR and CheXpert benchmarks, the RACER family achieved competitive or best-in-class budget-swept performance (measured via macro AURSBAC), proving its utility under real-world annotation noise and finite context constraints.
Limitations and Future Work
Despite its strong performance, RACER has a few notable limitations:
- Geometry Dependence: The framework relies heavily on an informative query-context embedding geometry. If the underlying image representation fails to organize examples by the factors driving expert competence, local role-relative pooling cannot recover the correct competence law.
- Finite Context Constraints: When the number of classes $K$ is exceptionally large, the expected same-role context support ($B/K$) shrinks, making local estimation noisy.
- Future Directions: Promising avenues include exploring approximate nearest-neighbor context retrieval for massive datasets, extending the framework to hierarchical multi-label taxonomies, and evaluating it across a broader range of non-medical deployment scenarios.
Summary updated