TL;DR
Standard conformal prediction (CP) for LLMs assumes that the model's configuration stays the same between calibration and deployment. This paper reveals "inference-time model shift"—where changes in quantization or temperature break uncertainty guarantees—and proposes robust new calibration methods to keep LLMs reliable in production.
What Problem Does This Paper Address?
Conformal Prediction (CP) is a powerful framework that provides rigorous statistical guarantees on model uncertainty. For example, it can produce a "prediction set" that is mathematically guaranteed to contain the correct answer 90% of the time. This guarantee relies on a core assumption called exchangeability: the model must behave the same way on the data used to calibrate it as it does on the data it sees in the real world.
While researchers often worry about dataset shift (the world changing), this paper identifies a more insidious problem: inference-time model shift. In real-world deployment, engineers often change a model's configuration—such as using quantization to save memory or adjusting "temperature" to control randomness—after it has been calibrated. The authors prove that these system-level choices alter the distribution of confidence scores, effectively breaking the statistical "contract" of CP and causing the promised 90% reliability to vanish.
Methods
The authors formalize the theory of inference-time model shift and conduct a massive empirical audit across 8 language models (including Llama 3.2, Mistral, and Gemma 2) and 12 different inference configurations (mixing quantization levels like fp16 and nf4 with temperatures from 0.5 to 2.0).
To address the reliability drops they discovered, they introduced and tested two robust calibration baselines:
- MAXTHR (Max-Threshold): A conservative approach that calculates the necessary threshold for every possible configuration and uses the largest one. It's safe but can lead to larger, less precise prediction sets.
- LATENT-ROBUST: A more sophisticated two-stage method. It uses the model's output "signature" (like the size of candidate prediction sets) to infer which configuration is likely active at test time and applies a tailored, adaptive threshold.
Datasets
The study evaluates these effects on four finite-label benchmarks, which allow for a controlled comparison of predictive distributions:
- MMLU-PRO: A robust and challenging multi-task understanding benchmark.
- CommonsenseQA: A question-answering task targeting everyday knowledge.
- DBPedia-14: A large-scale topic classification dataset.
- MobileActions: A dataset for predicting on-device function calling.
Key Insights and Ideas
- The "Matched Protocol" is a Trap: Most current evaluations use the same settings for calibration and testing. The authors show this is a "matched-only" illusion that hides massive potential failures in real-world serving environments.
- Configuration is a First-Class Variable: Inference settings like quantization are not just implementation details; they are as fundamental to model performance as the dataset or the architecture itself.
- Heterogeneity is the Rule: The impact of shift is not uniform. A configuration change that is benign for one model might cause another's reliability to collapse. There is no simple "monotone" rule (e.g., lower temperature isn't always "easier").
- Low Temperature is High Risk: Mismatches are often most severe in low-temperature regimes, which are ironically the settings most often used for high-stakes, "reliable" applications.
Key Results
The findings are a wake-up call for LLM deployment. As shown in Figure 1, while a "matched" model perfectly hits its 90% target, mismatched calibration can see coverage plummet to nearly 50%.
Across all datasets, standard "matched" evaluation (blue bars in Figure 2) is consistently over-optimistic. The "Mismatch worst" (green bars) shows that without robust calibration, the promised 90% guarantee can fail significantly in practice.
Crucially, the proposed LATENT-ROBUST and MAXTHR methods (brown and purple bars) successfully restored target coverage across almost every test case, proving that it is possible to maintain reliability even under configuration uncertainty.
Limitations and Future Work
- Characterization vs. Mechanism: The paper identifies that these shifts happen but does not yet provide a mechanistic theory to predict which configuration changes will cause under- or over-coverage for a specific model.
- Discrete Grids: The audit used a finite grid of configurations. Future work is needed to address continuous temperature variations or temporal drift in serving policies.
- Open-Ended Generation: The study focused on classification-like tasks. Extending these robust guarantees to free-form text generation remains a significant open challenge.
Summary updated