SHIFT-JEV: Certified Abstention and Gain-Gated Continual Adaptation for LLM-Based Decision Models under Distribution Shift

Muhammad Atif Saeed, Azka Atiq, Adarsh Kumar, Akhtar Jamil, Uzair Iqbal, Roland AydinarXiv 2026 · arXiv

TL;DR

SHIFT-JEV is a new reliability framework for LLMs acting as decision-makers, designed to handle the "silent killer" of machine learning: distribution shift. It combines certified abstention (refusing to answer when risk is too high) with a "gain-gated" adaptation strategy that ensures model updates actually improve performance without breaking what already works.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly used as "decision models"—systems that choose from a fixed set of options (e.g., routing a support ticket or grading a response). While they are powerful, they suffer from a major flaw: they are often "confidently wrong" when the data they see in production differs from their training data.

This "distribution shift" comes in many flavors:

  • Domain Shift: New topics or phrasing.
  • Class-Prior Shift: Changes in how often certain options are the correct answer.
  • Emerging Classes: New categories the model wasn't trained for.
  • Candidate-Set Shift: Changes to the list of options itself (reordering, rewording, or removing the right answer).

Current methods for providing risk guarantees (like conformal prediction) often fail when these shifts occur. The authors aim to create a system that can monitor these shifts, abstain from risky decisions to keep error rates within a budget, and safely adapt to new data over time.

Methods

The authors propose SHIFT-JEV, a three-stage pipeline added to the "LLM-as-Jev" interface (a method for getting calibrated probabilities from LLMs).

Overview of the SHIFT-JEV pipeline showing the three stages: scoring, reliability monitoring, and gated adaptation.
  1. Stage 1: Efficient Scoring. The model scores all possible options in a single forward pass using a prefix-free trie and a tree-shaped attention mask.
  2. Stage 2: The Reliability Layer.
    • Risk Scoring: Instead of just looking at the highest probability, it learns a risk score based on uncertainty, formatting signals (how much the model "wants" to output a valid option), and consistency (whether the answer changes if you rotate the options).
  3. Certified Abstention: It uses a statistical framework called Learn-then-Test (LTT) to set thresholds. If the risk score is too high, the model abstains.
  4. Shift Detection: It monitors the stream of decisions using drift detectors and a novel $\chi^2$ test on predicted labels to catch shifts that embedding-based detectors miss.
  5. Stage 3: Gain-Gated Continual Adaptation. When new labeled data arrives, the model fine-tunes using LoRA with "KL anchors" to prevent forgetting old knowledge. Crucially, an Expected-Gain Gate uses a sign test to verify the update significantly improves performance on new data without degrading old performance by more than 5 points.

Datasets

The authors introduced a comprehensive shift benchmark:

  • In-Distribution (ID): AG News, SST-2, BoolQ, and ARC-Challenge.
  • Proxy Shifts (for calibration): SciQ and Emotion.
  • Shift Benchmark: 10 environments covering domain shifts (Banking77, Twitter-Fin, MMLU), class-prior shifts, emerging classes (CLINC150), and candidate-set perturbations (rotation, paraphrasing, and removing the gold answer).
  • Models Tested: Qwen3-0.6B, Qwen3-4B, and Phi-4-mini.

Key Insights and Ideas

  • Label Shift is Invisible to Embeddings: One of the most striking findings is that "class-prior shift" (where the frequency of labels changes) is invisible to standard confidence and embedding-based detectors. The authors' $\chi^2$ test on predicted labels caught 95–99% of these shifts.
  • Uncertainty is Multi-Faceted: A simple "max probability" score is a poor proxy for risk under shift. Incorporating signals like the margin between the top two choices and the total number of options leads to much more robust abstention.
  • Certification Requires Re-verification: Guarantees of safety (risk control) are not "set and forget." They must be re-verified for each new deployment regime.
  • The Power of "No": In continual learning, the "Expected-Gain Gate" acts as a vital safeguard. It prevents the model from adapting to noisy or unhelpful data, which actually improved long-term stability and reduced "forgetting."

Key Results

  • Calibration: Fine-tuning with KL anchors and temperature scaling reduced calibration error under shift from 0.289 to 0.087.
  • Risk Control: At a target error rate of 10%, SHIFT-JEV's certified thresholds were violated in only 3–5 out of 27 tests, significantly outperforming standard conformal prediction sets (which failed in 9–14 tests).
  • Detection Performance: The system's detectors are highly sensitive to severe shifts, often triggering within just a few items.
Detection rate against shift severity for different detectors. The predicted-label test is uniquely capable of catching class-prior shifts.
  • Continual Learning: On the Qwen3-0.6B model, SHIFT-JEV reached 70.5% accuracy, outperforming standard baselines like Experience Replay, Learning without Forgetting (LwF), and EWC.

Limitations and Future Work

  • Small Sample Fragility: The "Learn-then-Test" certification is sensitive to sample size. With only 400 items for calibration, the system sometimes becomes overly cautious and abstains on everything to meet strict (5%) risk targets.
  • Conservative Gating: On larger, stronger models (like Qwen3-4B), the adaptation gate was sometimes too conservative, rejecting updates that offered small but real improvements because the statistical power was too low.
  • Feature-Rich Risk Models: For the frozen Phi-4-mini model, adding more signals actually increased realized risk, suggesting that complex risk models need to be carefully validated for each specific model backbone.
  • Future Directions: The authors suggest exploring larger per-regime calibration sets and label-shift-aware threshold corrections to further refine the reliability layer.
Table 1: Positioning. ✓: addressed; ◦: partially; —: not addressed.

Summary updated

Abstract

Large language models (LLMs) can act as decision models that return a probability distribution over a fixed set of options, but deployed decision models face domain, label-prior, emerging-class and candidate-set shift. We study how such models fail under shift and propose SHIFT-Jev, a monitoring, abstention and continual-adaptation procedure for them. It scores the risk of each decision from uncertainty, format and option-consistency signals, certifies abstention thresholds with Learn-then-Test, monitors the decision stream with drift and label-shift detectors, and accepts a model update only when an expected-gain gate finds a significant held-out improvement without loss on earlier regimes. On a benchmark of ten shifted test environments with Qwen3-0.6B and Qwen3-4B (three seeds each) and Phi-4-mini, fine-tuning with KL anchors and temperature scaling lowers calibration error under shift from 0.289 to 0.087. At a target risk of 10%, a learned uncertainty score with a certified threshold exceeds the target in 3 to 5 of 27 environment-seed tests, against 5 to 15 for a max-probability score with the same certification and 9 to 14 for conformal prediction sets; drift-based routing changes little. A chi-squared test on predicted labels detects 95 to 99% of class-prior shifts that confidence- and embedding-based detectors miss. In continual adaptation, SHIFT-Jev reaches 70.5% accuracy on Qwen3-0.6B, above replay, LwF, EWC and fine-tuning baselines in every seed, whereas on Qwen3-4B its conservative gate leaves it 0.4 to 1.4 points behind them.
BibTeX
@misc{saeed2026shift,
  author = {Muhammad Atif Saeed and Azka Atiq and Adarsh Kumar and Akhtar Jamil and Uzair Iqbal and Roland Aydin},
  title = {{SHIFT-JEV: Certified Abstention and Gain-Gated Continual Adaptation for LLM-Based Decision Models under Distribution Shift}},
  year = {2026},
  eprint = {2610.34329},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34329},
}
Figure