Conditional Calibration Contracts under Coupled Load: A Branch-Aware Audit of Switched Inference

Juan S. SantillanaarXiv 2026 · arXiv

TL;DR

This paper presents a rigorous audit of "switched inference" systems—architectures that dynamically toggle between large (teacher) and small (student) models to handle bursty loads. The authors find that these systems often fail to meet latency SLAs and suffer from "conditional calibration" issues, where a model's confidence scores become unreliable because the router only sends it a biased subset of traffic.

What Problem Does This Paper Address?

In modern machine learning deployment, "elastic inference" is a popular way to save costs: you use a small, cheap model for easy or low-priority requests and switch to a large, expensive model when the load is high or the input is difficult.

However, there is a hidden trap: selection bias. When a router decides which model sees which data, it changes the "population" each model handles. A model that is perfectly calibrated (meaning its 90% confidence actually corresponds to 90% accuracy) on a general dataset might become wildly uncalibrated on the specific, "hard" subset a router sends to it.

The authors argue that we shouldn't treat calibration as a static property of a model. Instead, they propose Conditional Calibration Contracts: versioned, per-cell budgets that must be validated against the specific traffic a model actually receives under a given routing policy.

Methods

The authors developed a diagnostic framework called CA-MHD (Conditional Audit of Multi-Head Distillation) to stress-test these systems. Key components include:

  • Load-Coupled Distillation (LCD): A training regime where the student model is trained not just to mimic the teacher, but to specifically handle the types of loads it will see in production.
  • Schmitt Router: A routing policy inspired by thermostats, using hysteresis (a "deadband") to prevent the system from "chattering" or switching models too frequently, which carries its own computational cost.
  • Discrete-Event Simulator (DES): A simulator that uses hardware service tables to model realistic queueing behavior and p99 latency under bursty arrival rates.
  • Kernel Correctness Gap ($H_\kappa$): A specialized metric used to measure calibration, focusing on the residual difference between predicted confidence and actual correctness.

Datasets

The study introduces CUL-Bench (Coupled Load Benchmark), a synthetic diagnostic setting.

  • It consists of 40 "cells" representing different combinations of training seeds and levels of "content-load coupling" ($\gamma$).
  • The coupling parameter $\gamma$ controls how much the difficulty of the task correlates with the arrival rate of requests, simulating real-world scenarios where a "burst" of traffic might also be more difficult to process.

Key Insights and Ideas

  • The "Teacher" Identity Crisis: The authors point out that in many distilled systems, the "teacher" branch isn't actually the original teacher. Because distillation often updates a shared feature trunk, the wide branch's performance can degrade compared to the standalone original model.
  • Calibration is Conditional: You cannot certify a router's performance based on a "teacher-always" trajectory. You must audit the model on the selected support—the specific slice of data the router chooses for it.
  • Routing Isn't a Silver Bullet: One of the most surprising "aha moments" is that routing often fails to provide a clear Pareto gain. In many cases, just using the student model alone resulted in better calibration and similar latency violations compared to a complex switching policy.
Figure 1: Campaign summaries versus 𝛾: medians and descriptive seed IQRs, not standard errors or confidence intervals. Left: worst nonempty served-stratum 𝐻𝜅(log scale); right: SLA violation (symlog). Plot label “smoothECE” is a legacy identifier for 𝐻𝜅.

Key Results

  • SLA Failures: Despite the complexity of the routing, student p99 latency remained above the 1.5 ms target in the simulation.
  • Ordinal Accuracy: The deployed "wide" branch actually had lower ordinal accuracy than the original teacher in all 40 test cells.
  • Negative Result for Routing: In 27 out of 40 scenarios, simply serving with the student model alone resulted in a lower historical ordinal gap than using the router.
  • Switching Efficiency: While the Schmitt router successfully reduced the number of model switches (typically to just 4 per horizon compared to hundreds in naive baselines), this efficiency didn't translate into meeting the p99 latency SLA.
Figure 3: Campaign transitions versus violation at 𝛾= 0.5. Many fewer transitions do not establish a latency Pareto gain: nominal B4 switch time is 0.057–0.211% of the horizon across 𝛾, and the switched violation tail remains important.

Limitations and Future Work

The authors are refreshingly honest about the limitations of their study:

  • Synthetic Environment: The results are based on a simulator. While the simulator uses hardware-validated service tables, it is not a live production environment.
  • Metric Bias: The historical calibration metric used ($H_\kappa$) has known implementation biases and does not inherit the theoretical guarantees of newer metrics like SmoothECE.
  • Unsolved Challenges: The paper concludes that creating a truly "validated prospective contract"—a guarantee that a system will stay calibrated and meet SLAs under future, unseen bursty loads—remains an open research problem. Future work needs to focus on independent budgets for different model branches and better validation of "selected-support" performance.

Summary updated

Abstract

Bursty serving changes both queue states and the population selected for each model. We study conditional calibration contracts as a specification to be validated, rather than a certified property of an existing router. CA-MHD combines load-coupled distillation, closed-loop replay and a cost-informed Schmitt router in a discrete-event simulator with a hardware service table. Across 40 cells (ten training seeds and four content-load coupling levels), we distinguish the original teacher, frozen wide heads on an adapted trunk, and the narrow student. A common teacher-trajectory audit has median absolute student-teacher deficit 0.06 percentage points, but does not certify the student-selected deployment support. Historical results use a kernel correctness gap, not the published SmoothECE; we separately prove selected-branch composition for a residual functional at a common fixed scale. The deployed wide branch has lower ordinal accuracy than the original teacher in all 40 cells. Same-checkpoint student-only serving has lower historical ordinal gap in 27/40 cells; routing also exposes violation tails up to 17.5 percent. Student p99 latency remains above the 1.5 ms target. Retrospective temporal recalibration improves the ordinal gap in 39/40 cells, but causal delayed-label availability is unverified and no incremental per-mode benefit is detected after multiplicity adjustment. An in-sample CUSUM detects 30/40 checkpoint changes at five historical false alarms, with no prospective false-alarm guarantee. These findings support a branch-aware audit and a scoped negative result, not demonstrated routing superiority. Independent budgets, selected-support validation and net utility under a satisfied SLA remain open.
BibTeX
@misc{santillana2026conditional,
  author = {Juan S. Santillana},
  title = {{Conditional Calibration Contracts under Coupled Load: A Branch-Aware Audit of Switched Inference}},
  year = {2026},
  eprint = {2611.02121},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02121},
}
Figure