RACER: Role-Aligned Competence Estimation for Human-AI Routing

Joshua Strong, Emma Sun, Alexander Capstick, Pramit Saha, Cheng Ouyang, J. Alison NoblearXiv 2026 · arXiv

TL;DR

RACER (Role-Aligned Competence Estimation for Routing) is a novel framework for population-adaptive "learning to defer" (L2D) that dynamically decides whether an AI or an unseen human expert should handle a given query. By estimating expert competence on a true probability scale relative to candidate class roles rather than absolute class identities, RACER achieves instance-dependent routing while remaining robustly invariant to arbitrary class relabellings.

What Problem Does This Paper Address?

In high-stakes fields like healthcare, AI models shouldn't always make the final call; instead, they should know when to defer to a human expert. Traditional Learning to Defer (L2D) systems assume a fixed, known expert during training. However, real-world deployments face rotating clinicians, newly hired specialists, or changing shifts. This requires population-adaptive L2D, where the system must infer the competence of an unseen expert at test time using only a small "context set" of their past decisions.

Prior approaches suffer from a strict dichotomy:

  1. Expressive but fragile methods (e.g., L2D-Pop): These use neural context encoders to route instance-by-instance, but they expose absolute class coordinates. This allows the network to learn brittle shortcuts tied to specific class indices or names rather than transferable expert skills.
  2. Robust but rigid methods (e.g., Identity-Free Deferral - IFD): These enforce invariance to class names by building classwise competence profiles. However, their estimates are constant within each class, failing to capture instance-level expert specialization (e.g., an expert who is great at typical cases of a disease but struggles with atypical ones).

RACER bridges this gap, achieving both query-dependent adaptation and identity-free transfer simultaneously.

A conceptual comparison highlighting how RACER bridges the gap between latent routing margins (L2D-Pop) and query-independent classwise profiles (IFD).

Methods

RACER separates classification, expert-competence estimation, and routing into distinct steps. Instead of mapping an expert's context to an unconstrained latent embedding, RACER estimates a posterior-predictive competence functional $\Gamma(x, y, C_e) \in [0, 1]$. This functional represents the probability that an expert (characterized by context $C_e$) will be correct on a query $x$, assuming the true class role is $y$.

The framework introduces a same-role kernel pooling primitive:

  • It computes the cosine similarity between the query $x$ and context examples.
  • It aggregates similarity masses only for context examples where the expert's true label matches the candidate role $y$.

The authors propose two main instantiations:

  • Racer-KNN: A transparent, nonparametric approach that takes a uniform average of the nearest same-role context neighbors.
  • Racer-kernel (Default): A neural approach that feeds role-relative summaries (such as local kernel statistics, posterior margins, ranks, and query uncertainty) into a small, role-shared Multi-Layer Perceptron (MLP).

To ensure these estimates behave like true probabilities, the competence head is trained using a strictly proper scoring rule (binary cross-entropy) with stop-gradients blocking routing-loss gradients from corrupting the competence parameters.

Datasets

The authors evaluated RACER on both controlled synthetic environments and real-world clinical datasets:

  • PathMNIST: A histopathology dataset used for a context-scaling study with simulated experts exhibiting hidden within-class subtype specializations.
  • CIFAR-100: Used to stress-test class cardinality ($K=20$ and $K=100$) and expert-profile permutations.
  • VinDr-CXR: A real-world, multi-radiologist chest X-ray dataset used to evaluate routing performance under realistic expert variability and annotation noise.
  • CheXpert: A multi-expert human-AI chest radiography benchmark containing annotations from multiple human readers and external pretrained AI systems.

Key Insights and Ideas

  • Role-Relative vs. Absolute Coordinates: By evaluating competence relative to a candidate class role (e.g., "how well does the expert perform when the true label is $y$?") rather than absolute class indices, RACER inherently satisfies coherent class-relabelling invariance. If you rename the classes, the routing decisions remain semantically identical.
  • Probability-Scale Calibration: Unlike methods that output arbitrary latent routing scores, RACER targets the true conditional probability of expert correctness. This explicit calibration is crucial for real-world triage, multi-expert routing, and threshold selection under varying operational budgets.
  • Resolving Within-Class Bias: Classwise profiles (like IFD) fail when an expert's skill varies within a class due to image quality, subtype variations, or local morphology. RACER's query-conditioned kernel pooling successfully extracts local context evidence to resolve this within-class bias.

Key Results

  • Context Scaling Wins: In the PathMNIST study under hidden subtype dependence, Racer-kernel demonstrated a clear context-scaling trend. While classwise profiles plateaued due to within-class bias, Racer-kernel converted larger context sets into substantial routing utility gains, crossing the classifier baseline at $B/K \approx 5.6$.
  • Robustness to Cardinality: On the CIFAR-100 benchmark, Racer-kernel was the only evaluated method to achieve a positive aggregate out-of-distribution (OOD) routing gain over the classifier baseline across both large and small label spaces ($K=20$ and $K=100$).
  • Clinical Relevance: On the real-radiologist VinDr-CXR and CheXpert benchmarks, the RACER family achieved competitive or best-in-class budget-swept performance (measured via macro AURSBAC), proving its utility under real-world annotation noise and finite context constraints.

Limitations and Future Work

Despite its strong performance, RACER has a few notable limitations:

  • Geometry Dependence: The framework relies heavily on an informative query-context embedding geometry. If the underlying image representation fails to organize examples by the factors driving expert competence, local role-relative pooling cannot recover the correct competence law.
  • Finite Context Constraints: When the number of classes $K$ is exceptionally large, the expected same-role context support ($B/K$) shrinks, making local estimation noisy.
  • Future Directions: Promising avenues include exploring approximate nearest-neighbor context retrieval for massive datasets, extending the framework to hierarchical multi-label taxonomies, and evaluating it across a broader range of non-medical deployment scenarios.

Summary updated

Abstract

Learning to defer asks a predictive system when to act autonomously and when to defer to a human expert. Population-adaptive deferral extends this problem to unseen experts using a small context set of expert behavior. Neural context encoders such as L2D-Pop can be query-dependent, but may learn routing shortcuts tied to absolute class coordinates. Identity-Free Deferral (IFD) removes such shortcuts through role-indexed classwise competence profiles, but its estimates are constant within each class and cannot capture instance-level expert specialization. We propose RACER---Role-Aligned Competence Estimation for Routing---a role-relative framework for estimating an unseen expert's competence from context. RACER estimates the posterior-predictive probability that the expert is correct on a query under each candidate class role, then combines these estimates with the model posterior to obtain the Bayes-relevant expert-correctness probability. Nonparametric and neural kernel-pooling estimators use candidate-role relations, shared aggregation, and symmetric summaries, excluding absolute class-identity channels. We prove coherent class-relabelling invariance, derive a Bayes-aligned deferral surrogate, and give a plug-in regret bound relating routing regret to classifier and competence-estimation error. On controlled synthetic benchmarks, including a PathMNIST histopathology context-scaling study with simulated experts, RACER benefits from additional context under hidden subtype dependence and gives the strongest aggregate performance on a separately sampled unseen-expert split in the CIFAR-100 synthetic experiments. On the radiologist and human--AI chest-radiography benchmarks (VinDr-CXR and CheXpert), the RACER family is competitive or best in budget-swept deferral, with calibration results varying across metrics and datasets.
BibTeX
@misc{strong2026racer,
  author = {Joshua Strong and Emma Sun and Alexander Capstick and Pramit Saha and Cheng Ouyang and J. Alison Noble},
  title = {{RACER: Role-Aligned Competence Estimation for Human-AI Routing}},
  year = {2026},
  eprint = {2609.21953},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.21953},
}
Figure