A clinical question-answering system should refuse to answer when it is likely to be wrong, because a wrong answer costs more than sending the question to a human. The standard recipe has two steps. First, measure how confident the model is on held-out questions. Second, pick the confidence cutoff below which the model abstains, choosing it to minimize expected cost. This recipe quietly assumes that the confidence seen during tuning reflects the confidence the model will have on real questions.
On widely used benchmarks that assumption fails. Some test questions leaked into training data. Others share telltale phrases with the training set. In both cases the model looks confident for the wrong reason, so the cutoff tuned on that data accepts too much once deployed. The clearest case in the file is EHRSQL, a benchmark for turning clinicians' questions into database queries over electronic health records, where models must also refuse unanswerable questions. Reworded or post-cutoff questions are the natural probe of this failure. The proposal is to make the model's confidence on the original wording agree with its confidence on a meaning-preserving rewording, both when training and when choosing the cutoff. The test is whether a cost-optimal cutoff chosen on ordinary validation data then still holds on shortcut-free test questions.
Core Claim
The claim has a measurement part and a method part. The measurement part is that lexical shortcuts and contamination move the empirically best reject threshold away from the threshold chosen on validation data, and that this displacement is predictable. The predictor is the gap between the model's confidence on an original question and on its rewording, the same paired gap that PaCoST uses to audit whole benchmarks for contamination. The method part is that turning this audit signal into a training penalty, alongside focal loss, shrinks the displacement more than three cheaper fixes: post-hoc rescaling of probabilities, plain focal loss, or simply adding rewordings to the training data. Focal loss here means a loss that down-weights examples the model already gets right easily.
The cost-optimal threshold comes from Rethinking and recomputing the value of machine learning models. That paper derives the threshold and shows that net value turns negative once calibration breaks under distribution shift. A model acts autonomously when its top-class probability exceeds
$$\tau = \frac{C_{\text{error}} - C_{\text{reject}}}{C_{\text{error}} - C_{\text{correct}}},$$
where $C_{\text{error}}$, $C_{\text{reject}}$ and $C_{\text{correct}}$ are the costs of a wrong answer, an abstention and a right answer. This rule is only as good as the probabilities fed into it. On the Calibration of Multiclass Classification with Rejection shows that thresholding the top probability of a well-calibrated classifier beats training a separate rejector. That result supplies our design choice: train one model, then threshold it post hoc. Everything therefore rests on whether the probabilities can be trusted on shortcut-free inputs.
Why EHRSQL Is the Right Testbed
EHRSQL has the domain gap and the shortcut problem in one place. The original benchmark paper, EHRSQL, reports two things. A general-domain text-to-SQL parser reaches only 4.7% accuracy on it. A T5-base model that abstains above an entropy percentile reaches answerability F1 above 90%. Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL then shows that this reliability is largely illusory: phrases such as "phone number" appear almost only in unanswerable questions. Its numbers make the point sharply:
- A T5 model with an n-gram filter scores 93.2 F1 on the original split but 1.6 F1 on a re-partitioned "New Split", where high-bias phrases are moved to test.
- Stronger methods drop from about 93 to about 62 F1 on the same New Split.
That paper fixes the evaluation by changing the split. It does not change how models are trained, and it does not ask what happens to a cost-tuned cutoff. The shared-task system LTRC-IIITH at EHRSQL 2024 used a hand-picked log-probability cutoff of −0.4 with no calibration and placed 6th. We treat shortcut leakage between the train and test splits as a small, fully observable version of benchmark contamination: the model is rewarded for recognizing surface form rather than reasoning about the database schema.
Research Program
For every training and validation question $x$, we build a rewording $x'$ that keeps its meaning and its answerability label but avoids phrases the New Split analysis flags as high-bias. Answerable questions can draw on the SQL-aware augmentations in Text2SQL-Flow, which contains 11,721 EHRSQL-derived pairs. Unanswerable questions need a language-model rewrite that keeps the requested attribute absent from the schema, for example "contact line for the attending" instead of "phone number". We check a sample of these rewrites by hand, because a careless rewrite could turn an unanswerable question into an answerable one.
The model is trained on each pair with
$$L(x,x') = \mathrm{FL}(x) + \mathrm{FL}(x') + \lambda\,\big[\hat p(x) - \hat p(x')\big]_+ .$$
Here $\mathrm{FL}$ is focal loss, $-(1-p_t)^{\gamma}\log p_t$, where $p_t$ is the probability of the true label and $\gamma$ is the focusing strength. $\hat p$ is the model's top-class probability, and $[\cdot]_+$ keeps only positive values. The hinge term penalizes the model whenever it is more confident on the original wording than on the rewording. This penalty is PaCoST's audit statistic used as a training signal, which is precisely the application that PaCoST itself leaves open.
Focal loss plays a supporting role for two reasons. First, unanswerable questions are a minority class, and we will measure the actual ratio. Second, Calibrating Deep Neural Networks using Focal Loss shows that focal loss resists the overconfidence cross-entropy develops and holds calibration better under shift.
We do not assume focal loss helps on its own. Revisiting Reweighted Risk for Calibration reports focal loss at 5.83% calibration error (ECE, the average gap between confidence and accuracy) against 2.62% for cross-entropy on CIFAR-10. That is evidence focal reweighting can hurt calibration. There is also a specific reason it cannot fix shortcuts: a shortcut-solved question looks easy, so focal loss simply stops training on it and leaves its inflated confidence in place. The hinge term is the part that targets that inflation.
The threshold is then chosen with the cost formula, applied to probabilities rescaled on the reworded validation set. This makes paired rewording a tool both for training and for selecting the cutoff. We run this setup with T5-base, matching the EHRSQL baseline, and with SQLCoder-7b-2, matching LTRC-IIITH. Abstention is applied both to the answerability classifier and to the generation log-probability gate.
A secondary study tests whether the same displacement appears with general LLMs on contaminated benchmarks. We take open models on MMLU professional and medical subsets, where PaCoST flagged contamination, and on WMDP, which PaCoST used as a clean reference. For each, we tune a selective-answering cutoff on original wording and evaluate it on reworded items. We then ask whether per-benchmark displacement tracks PaCoST's paired gap. This probes the conjecture in Rethinking and recomputing the value of machine learning models that GPT-3 looked robust out of distribution only because few public benchmarks are truly out of distribution.
Baselines
Naive reference: shortcut-tuned cross-entropy. This is a cross-entropy model whose threshold is tuned on the original-split validation set. It reproduces the EHRSQL protocol of EHRSQL, including the LTRC-IIITH fixed −0.4 log-probability gate. It is the do-nothing point, and we expect it to show the largest displacement. For reference, LTRC-IIITH reports a 6th-place leaderboard finish but no numeric score.
Strongest existing number: the New Split comparison. The strongest reported result on the honest evaluation is about 62 F1 for the best methods on the New Split, from Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL. We re-run those methods on the identical split. Beating them means two things together: higher unanswerable-detection F1 than the re-run baseline on the New Split, and lower realized cost at the same cost ratios.
Strongest calibration competitor: AdaFocal. AdaFocal adapts the focusing strength separately for each confidence range, using validation feedback. It reports 1.36% calibration error versus 17.17% for cross-entropy on CIFAR-100. No reported result was found on Lacuna for AdaFocal on EHRSQL. We run it with the same backbones, data, epochs and hyperparameter search budget, with its validation feedback drawn from either original or reworded validation data.
Ablations isolating the key ingredient. Two ablations isolate the hinge. The first is reworded-data augmentation with plain focal loss and no hinge. The second is the hinge with cross-entropy instead of focal loss.
Fairness and the success criterion. All arms see the same reworded pairs, the same number of gradient steps and the same tuning budget. The proposal counts as winning only if it reduces threshold displacement and realized cost relative to augmentation-only by more than seed-to-seed variance. Otherwise the conclusion is that rewording the data, not the loss, does the work.
Why not just…?
Why not just temperature-scale and tune the threshold on a shortcut-free validation set? This is the most serious threat. The New Split already keeps its validation set free of bias phrases, so a cutoff tuned there may transfer well. Two things limit it. First, Rethinking and recomputing the value of machine learning models finds that rescaling probabilities after training degrades under shift. Second, a single temperature cannot repair calibration errors that differ across confidence ranges, which AdaFocal documents. A further practical limit is that deployers rarely have a debiased split, whereas rewordings can be generated for any benchmark. We include New-Split-validation rescaling as an explicit arm. If it matches the full method on cost and displacement, the honest contribution narrows to the reworded-validation protocol.
Why not just add the rewordings to training data? This is also a real threat. Augmentation alone may remove the shortcut. The augmentation-only ablation settles it directly. Our reason to expect a gap is that augmentation raises accuracy on reworded questions without forbidding extra confidence on original phrasings. That extra confidence on original phrasings is precisely what moves the threshold, so we report displacement separately from accuracy.
Why not just use focal loss or AdaFocal, since they already calibrate? Their down-weighting treats shortcut-solved questions as easy and stops correcting them. Revisiting Reweighted Risk for Calibration also shows that focal loss can raise calibration error and that selective prediction depends on how confidences are ranked, not their scale. The AdaFocal arm tests this directly.
Why not just mask known shortcut phrases? Masking needs the shortcut list in advance. The n-gram filter's fall from 93.2 to 1.6 F1 shows how brittle list-based fixes are once phrasing changes. We still include phrase masking as a cheap arm.
Proposed Experiments
The main metric is threshold displacement. We compute the cost-optimal cutoff on validation data, compute the cutoff that actually minimizes cost on New Split test data, and take the difference. We report it alongside four other quantities:
- realized net cost at error-cost multiples $k$ of 2, 10 and 50, following Sayin et al.'s cost-ratio parameterization;
- the area under the risk–coverage curve;
- calibration error on original versus reworded test questions;
- unanswerable-detection F1 and execution accuracy on the New Split.
The first decisive check is whether displacement for cross-entropy is substantially nonzero on EHRSQL, and whether the paired confidence gap on validation predicts it. If displacement is negligible, the problem this proposal targets does not exist in this setting. The second check is whether the full objective beats augmentation-only on realized cost at high $k$, where overconfidence matters most. The third check, the cross-benchmark LLM study, asks whether displacement grows with PaCoST's gap across MMLU subsets and is near zero on WMDP. If it does, displacement extends beyond one benchmark to contaminated benchmarks generally. Every arm is run over five seeds, and differences are reported with confidence intervals.
References
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- Rethinking and recomputing the value of machine learning models
- On the Calibration of Multiclass Classification with Rejection
- EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records
- Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL
- LTRC-IIITH at EHRSQL 2024: Enhancing Reliability of Text-to-SQL Systems through Abstention and Confidence Thresholding
- Calibrating Deep Neural Networks using Focal Loss.
- Revisiting Reweighted Risk for Calibration: AURC, Focal Loss, and Inverse Focal Loss
- AdaFocal: Calibration-aware Adaptive Focal Loss