Two families of training recipes exist for systems where a classifier can either answer itself or hand the input to a human expert, and their published conclusions point in opposite directions. Work on abstention, meaning a model that may decline to answer, finds that training the predictor first and then fitting a separate "rejector" on its mistakes beats training both jointly. On SVHN this two-stage recipe nearly halves the error of the best joint surrogate. Work on learning to defer, meaning a model trained to route inputs to a human, argues the reverse. Only joint training lets the classifier stop spending capacity on cases the human already gets right, and focus instead on the human's blind spots. We propose to settle this disagreement with one quantity from learning theory and one simple algorithm.
The quantity is the minimizability gap: how much loss a restricted model family (say, a linear head on a frozen backbone) must pay compared with an unrestricted predictor, for a given training loss. Our claim is that the gap explains the conflict. If the model family is rich enough that the classifier's gap is near zero on the whole data distribution, there is nothing to gain from specializing, and the two-stage recipe's better optimization behavior wins. If the family is capacity-limited, the gap measured under a loss weighted by where the human errs can be much smaller than the gap under the plain loss. Joint training captures this difference and the standard two-stage recipe throws it away. We test this claim, and we test a hybrid that should recover most of the benefit: a two-stage method whose first stage trains the classifier on expert-error-weighted data. This hybrid keeps the guarantees of the two-stage recipe while still allowing specialization.
The Disagreement in the Literature
On the two-stage side, Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstention treats abstention as an extra label with a fixed cost. It proves non-asymptotic $\mathcal{H}$-consistency bounds and shows that joint ("single-stage") surrogates can miss a zero-error solution even when one exists in the model family. The two-stage scheme reduces abstention loss from 3.62% to 3.22% on CIFAR-10, from 10.40% to 9.54% on CIFAR-100, and from 1.61% to 0.93% on SVHN. Here, $\mathcal{H}$-consistency means a guarantee that driving down the training surrogate within the model family $\mathcal{H}$ also drives down the true 0-1 error within $\mathcal{H}$.
On the one-stage side, Consistent Estimators for Learning to Defer to an Expert, together with Wilder et al. and Nushi et al., argues that joint training is what produces complementary specialization. Two follow-up papers make joint training theoretically respectable. Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defer introduces a surrogate family that is both Bayes-consistent and realizable-$\mathcal{H}$-consistent, and shows that these two properties are independent in deferral. Mastering Multiple-Expert Routing extends both one-stage and two-stage guarantees to many experts.
None of these papers compares how the minimizability gap differs between the two recipes as model capacity shrinks. That comparison is the open question named in the research file's third gap. These papers motivate our question, and their surrogates supply our methods. The abstention result also acts as a constraint: any hybrid we build must keep the two-stage structure that avoids realizability failure.
Core Claim
Let $\ell$ be the multiclass cross-entropy and $\mathcal{H}$ the classifier family. Let $e(x)$ be the probability that the human errs on input $x$. The standard two-stage recipe trains the classifier to minimize $\mathbb{E}[\ell(h(x),y)]$ and pays that loss's minimizability gap $\mathcal{M}_{\mathcal{H}}(\ell)$. On the inputs the rejector keeps, however, team performance depends roughly on the classifier's loss where the human would fail. That is the weighted loss $\ell_e(h,x,y) = e(x)\,\ell(h(x),y)$, whose gap we write $\mathcal{M}_{\mathcal{H}}(\ell_e)$. We conjecture, and aim to prove for comp-sum surrogates, a team-level bound for each recipe. Both bounds share the same square-root growth term, which follows from the universal rate in A Universal Growth Rate for Learning with Smooth Surrogate Losses. The recipes differ through an additive gap term. For standard two-stage learning, this term is governed by $\mathcal{M}_{\mathcal{H}}(\ell)$. For joint learning and for our weighted two-stage hybrid, it is governed by $\mathcal{M}_{\mathcal{H}}(\ell_e)$ plus the rejector's own gap.
The prediction is testable. The advantage of joint training should track the difference $\mathcal{M}_{\mathcal{H}}(\ell)-\mathcal{M}_{\mathcal{H}}(\ell_e)$. This difference shrinks toward zero as capacity grows, which would explain why abstention papers using wide ResNets see no benefit from joint training. The weighted two-stage hybrid should match joint training's specialization while inheriting two-stage robustness to realizability failure.
Research Program
The theory part extends the two-stage abstention analysis to a first stage trained on expert-error-weighted losses. We will derive the bound above and characterize when the gap difference is strictly positive. One natural case is a linear family with expert errors concentrated in a region the family cannot fit jointly with the rest of the data.
The empirical part measures the gaps directly, which addresses the file's noted difficulty that gaps are hard to estimate for deep networks. On synthetic Mixture-of-Gaussians tasks with linear and small-MLP families, Bayes quantities are computable, so gaps are exact. On real data, we use a frozen backbone and vary the capacity of the trained head, from linear to a width-swept MLP. We estimate the "all functions" infimum with a much larger head trained to convergence on held-out data. This gives an upper estimate of each gap, which is sufficient for comparing recipes.
The hybrid algorithm runs in two steps. First, it estimates $e(x)$ with a small model fit to expert-correctness labels. Second, it trains the classifier on $\ell_{\hat e}$, then trains the rejector with the two-stage multi-expert surrogate from Mastering Multiple-Expert Routing. We also add a temperature that mixes $\hat e$ toward uniform. This interpolates between standard two-stage training (temperature fully uniform) and full specialization.
Baselines
The naive reference points are the classifier alone, the human alone, and a classifier trained normally that defers whenever its maximum softmax probability falls below a threshold. For the confidence baseline, Consistent Estimators for Learning to Defer to an Expert reports that joint deferral beats it by 1.2 points of system accuracy on CIFAR-10H. Any method we propose must also clear this margin.
The strongest existing competitor is the one-stage realizable surrogate of Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defer. It reports 92.20% system accuracy on HateSpeech against 91.60% for the cross-entropy deferral surrogate, and 76.77% coverage on CIFAR-10H. It is the strongest because it already trains jointly, so it already captures specialization. Our hybrid wins only if the two-stage structure adds something beyond that.
We also run two other existing recipes:
- the two-stage abstention and routing scheme of Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstention and Mastering Multiple-Expert Routing;
- for the multi-expert real-annotator sets, CL2DC from Coverage-Constrained Human-AI Cooperation with Multiple Experts, which reports 88.88% area under the accuracy–coverage curve on CIFAR-100 against 81.17% for the next method.
Fairness rules apply to all methods. Every method uses the same frozen backbone, the same head-capacity grid, and the same expert annotations, which the one-stage baselines also require during training. Each method gets an identical hyperparameter budget of 30 configurations per capacity level, selected on a shared validation split, with five seeds.
The key ablation sets the weighting temperature to uniform, which recovers plain two-stage training. A second ablation uses oracle $e(x)$ instead of the estimated $\hat e(x)$, to separate estimation error from the principle.
Beating the strongest baseline means the following:
- At small capacities, the hybrid matches or exceeds the one-stage realizable surrogate's system accuracy, with a statistically clear margin. The comparison is made at matched coverage, or on the accuracy–coverage area across coverage.
- On the realizable synthetic tasks, the hybrid keeps two-stage near-perfect accuracy where the one-stage surrogate does not.
Why not just…?
Why not just threshold the classifier's confidence? Confidence reflects the classifier's uncertainty and ignores where the human fails. When human error is uneven across inputs, this is provably the wrong signal. The measured gaps quoted above, and the "fairness trap" result in Consistent Estimators for Learning to Defer to an Expert, show this. We include the baseline mainly to calibrate effect sizes.
Why not just use the one-stage realizable surrogate everywhere? This is the serious threat. If Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defer already matches the hybrid at every capacity, the hybrid's practical value collapses, though the gap analysis still has explanatory value. Two features favor two-stage structure in practice. Two-stage training lets the rejector be retrained when experts change without touching the classifier, and the abstention results show single-stage optimization can underperform even with good losses. The capacity-sweep experiment decides this directly. If the hybrid shows no advantage over one-stage training at any capacity, we will report the hybrid as unnecessary and keep the diagnostic as the contribution.
Why not just make the model bigger? Our own theory predicts that at high capacity the gap difference vanishes and the recipes converge. That would be a confirmation of the theory, not a refutation. The regime that matters is deployment with frozen foundation-model backbones and small heads, which is exactly the setting Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defer leaves open. The capacity sweep shows where on the curve the advantage disappears.
Why not just reweight by expert error heuristically, without the theory? The weighting itself is simple. What the theory adds is a prediction of when weighting helps, namely in proportion to the measured gap difference. That prediction can fail, and the next section tests it.
Decisive Checks
The datasets are:
- synthetic Mixture-of-Gaussians tasks with exact gaps;
- CIFAR-10H and HateSpeech with real human labels;
- Chaoyang and NIH-ChestXray with multiple real annotators;
- CIFAR-100 with synthetic class-specialist experts, to vary how concentrated human errors are.
The metrics are system accuracy at matched coverage, the area under the accuracy–coverage curve, and the classifier's accuracy on the inputs it keeps.
The first decisive check concerns the gap difference. Across all capacities and datasets, the measured difference $\mathcal{M}_{\mathcal{H}}(\ell)-\mathcal{M}_{\mathcal{H}}(\ell_e)$ should rank-correlate with the system-accuracy advantage of one-stage over standard two-stage training. If this correlation is absent, the core claim is wrong.
The second check concerns realizability. On realizable synthetic tasks, the hybrid should keep near-100% system accuracy, the level Mastering Multiple-Expert Routing reports for its surrogate. Meanwhile, under capacity limits, it should match the one-stage baseline.
The third check concerns concentration of human errors. Making the synthetic experts' errors more concentrated should increase both the gap difference and the hybrid's benefit together. Making the errors uniform should erase both.
References
- Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstention
- Consistent Estimators for Learning to Defer to an Expert
- Learning to Complement Humans
- Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork
- Realizable $H$-Consistent and Bayes-Consistent Loss Functions for Learning to Defer
- Mastering Multiple-Expert Routing: Realizable $H$-Consistency and Strong Guarantees for Learning to Defer
- A Universal Growth Rate for Learning with Smooth Surrogate Losses
- Coverage-Constrained Human-AI Cooperation with Multiple Experts