Auditing Conformal Prediction for Language Models Under Inference-Time Model Shift

Achref Doula, Max MühlhäuserarXiv 2026 · arXiv

TL;DR

Standard conformal prediction (CP) for LLMs assumes that the model's configuration stays the same between calibration and deployment. This paper reveals "inference-time model shift"—where changes in quantization or temperature break uncertainty guarantees—and proposes robust new calibration methods to keep LLMs reliable in production.

What Problem Does This Paper Address?

Conformal Prediction (CP) is a powerful framework that provides rigorous statistical guarantees on model uncertainty. For example, it can produce a "prediction set" that is mathematically guaranteed to contain the correct answer 90% of the time. This guarantee relies on a core assumption called exchangeability: the model must behave the same way on the data used to calibrate it as it does on the data it sees in the real world.

While researchers often worry about dataset shift (the world changing), this paper identifies a more insidious problem: inference-time model shift. In real-world deployment, engineers often change a model's configuration—such as using quantization to save memory or adjusting "temperature" to control randomness—after it has been calibrated. The authors prove that these system-level choices alter the distribution of confidence scores, effectively breaking the statistical "contract" of CP and causing the promised 90% reliability to vanish.

Methods

The authors formalize the theory of inference-time model shift and conduct a massive empirical audit across 8 language models (including Llama 3.2, Mistral, and Gemma 2) and 12 different inference configurations (mixing quantization levels like fp16 and nf4 with temperatures from 0.5 to 2.0).

To address the reliability drops they discovered, they introduced and tested two robust calibration baselines:

  • MAXTHR (Max-Threshold): A conservative approach that calculates the necessary threshold for every possible configuration and uses the largest one. It's safe but can lead to larger, less precise prediction sets.
  • LATENT-ROBUST: A more sophisticated two-stage method. It uses the model's output "signature" (like the size of candidate prediction sets) to infer which configuration is likely active at test time and applies a tailored, adaptive threshold.

Datasets

The study evaluates these effects on four finite-label benchmarks, which allow for a controlled comparison of predictive distributions:

  • MMLU-PRO: A robust and challenging multi-task understanding benchmark.
  • CommonsenseQA: A question-answering task targeting everyday knowledge.
  • DBPedia-14: A large-scale topic classification dataset.
  • MobileActions: A dataset for predicting on-device function calling.

Key Insights and Ideas

  • The "Matched Protocol" is a Trap: Most current evaluations use the same settings for calibration and testing. The authors show this is a "matched-only" illusion that hides massive potential failures in real-world serving environments.
  • Configuration is a First-Class Variable: Inference settings like quantization are not just implementation details; they are as fundamental to model performance as the dataset or the architecture itself.
  • Heterogeneity is the Rule: The impact of shift is not uniform. A configuration change that is benign for one model might cause another's reliability to collapse. There is no simple "monotone" rule (e.g., lower temperature isn't always "easier").
  • Low Temperature is High Risk: Mismatches are often most severe in low-temperature regimes, which are ironically the settings most often used for high-stakes, "reliable" applications.

Key Results

The findings are a wake-up call for LLM deployment. As shown in Figure 1, while a "matched" model perfectly hits its 90% target, mismatched calibration can see coverage plummet to nearly 50%.

MMLU-PRO coverage drops significantly when calibration and deployment configurations don't match.

Across all datasets, standard "matched" evaluation (blue bars in Figure 2) is consistently over-optimistic. The "Mismatch worst" (green bars) shows that without robust calibration, the promised 90% guarantee can fail significantly in practice.

Coverage across different models and datasets. The robust baselines (MaxThr and Latent-robust) successfully restore the target 0.9 coverage.

Crucially, the proposed LATENT-ROBUST and MAXTHR methods (brown and purple bars) successfully restored target coverage across almost every test case, proving that it is possible to maintain reliability even under configuration uncertainty.

Limitations and Future Work

  • Characterization vs. Mechanism: The paper identifies that these shifts happen but does not yet provide a mechanistic theory to predict which configuration changes will cause under- or over-coverage for a specific model.
  • Discrete Grids: The audit used a finite grid of configurations. Future work is needed to address continuous temperature variations or temporal drift in serving policies.
  • Open-Ended Generation: The study focused on classification-like tasks. Extending these robust guarantees to free-form text generation remains a significant open challenge.

Summary updated

Abstract

Conformal prediction (CP) for language models is typically evaluated under a matched protocol, in which calibration and deployment use the same post-training inference configuration, such as quantization mode and inference temperature. In deployment, these choices often differ from those used at calibration to meet memory or throughput constraints or serving policies. We argue that this exposes a limitation in current CP evaluation protocols for language models. We formalize the underlying phenomenon as inference-time model shift: a change in the conformal score distribution induced by inference configuration rather than by the input--label distribution, and theoretically characterize how it breaks split-conformal exchangeability even when the task distribution and trained model are unchanged. We then evaluate its consequences across 8 language models, 4 finite-label benchmarks, and 12 inference configurations. We find that matched evaluation can substantially overstate deployment-time coverage, and that the effect is heterogeneous across models and datasets. We argue that inference configuration should be reported as a first-class experimental axis in CP evaluations for language-model systems.
BibTeX
@misc{doula2026auditing,
  author = {Achref Doula and Max Mühlhäuser},
  title = {{Auditing Conformal Prediction for Language Models Under Inference-Time Model Shift}},
  year = {2026},
  eprint = {2610.28937},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.28937},
}
Figure