When a language model reads a lab value and says "this patient is high-risk," it must be comparing the value against some threshold. In clinical medicine, the threshold is often the weakest part of the claim. The same marker gets different published cutoffs in different cohorts. The albumin-to-globulin ratio's cutoff ranges from 1.14 to 1.93 across twelve gastric-cancer studies (Wei et al.). A neutrophil-to-lymphocyte cutoff of 3.24 is simply the median of one hospital's cohort (Feng et al.). A model trained on web text has seen these numbers. If it answers lab-interpretation questions by recalling a paper's cutoff rather than the reference range the question actually supplies, its benchmark score measures memorized literature, not clinical reasoning. That is benchmark contamination at the level of a single number, and nothing in current contamination detection looks for it.
We propose to recover, for each model and each biomarker, the threshold the model is implicitly using. The procedure asks many cheap probe questions that vary the lab value, the stated reference range, the measurement unit, and the gestational or demographic context. It then combines the yes/no answers with Dempster-Shafer belief functions, a way of representing evidence that can assign weight to sets of possibilities and to "don't know," not just to single outcomes. In general this combination is exponentially expensive. Here the possible thresholds lie on an ordered line, and each probe answer rules out a contiguous range of them, so the combination stays polynomial. The output separates three things that a single accuracy number collapses: the model tracks the supplied reference (sound reasoning); the model locks onto a published cutoff (memorization); or the answers are inconsistent or uninformative (confusion or ignorance).
Core Claim
A model's implicit decision threshold for a biomarker can be identified from black-box probes, and its location relative to (a) the reference range stated in the prompt and (b) cutoffs published in the literature is a per-item contamination signal. This signal is invisible to accuracy-based detectors.
The belief-function representation earns its place for two reasons. It keeps "the probes disagree," measured by Dempster's conflict mass, separate from "the probes say nothing," measured by mass left on the whole frame. It also does so without a prior over thresholds, which matters when only a few dozen probes per biomarker are affordable.
The computational point is simple. Let the frame be a grid of $k$ candidate thresholds $\tau_1 < \dots < \tau_k$. A probe at value $v$ answered "abnormal" is consistent only with thresholds below $v$, which is an interval. We discount each probe by a reliability $\alpha$ estimated from the model's agreement with itself under paraphrase. The probe then becomes a mass function with exactly two focal sets, the interval and the whole frame $\Theta$. Because intervals are closed under intersection, the combined mass function never holds more than $k(k+1)/2$ focal intervals. Combining $n$ probes therefore costs $O(nk^2)$ rather than the $O(2^k)$ of unrestricted Dempster combination (Chen et al.). Two-sided reference ranges give rectangle-shaped focal sets at $O(nk^4)$. Joint thresholds over $d$ markers grow as $k^{2d}$. That joint case is where the tractability limits documented by Martin et al. bite again, and we address it below.
Relation to Prior Work
Three lines of work motivate the proposal.
The first is the biomarker literature, which shows that cutoffs are cohort artifacts:
- Feng et al. used a cohort-median NLR that predicted progression-free survival but not overall survival.
- Wei et al. documented AGR cutoffs scattered under 76.8% heterogeneity.
- Zuo et al. reported a GNRI cutoff of 97 and flagged the lack of ethnicity-specific standards.
- Luo et al. showed that angiogenic markers need gestational-age percentile normalization (PlGF%) rather than a fixed number.
- Wan et al. showed that assay platforms disagree on identical samples.
These papers give us both the known "memorizable" cutoffs and the perturbation axes: unit, assay, and gestational age.
The second is clinical LLM evaluation, which treats lab values as fixed facts. LiveClin avoids leakage by refreshing cases from recent case reports, at a cost of over 1,700 physician-hours per refresh. It scores multiple-choice accuracy and never asks whether a correct lab interpretation came from a memorized cutoff. Revisiting MIMIC-IV verbalizes ICU labs through fixed templates and finds that zero-shot LLMs are near chance. Our probes are a cheap complement to both: perturbing the threshold is far cheaper than finding new cases.
The third is contamination detection, which supplies the evaluation logic we extend:
- ConStat defines contamination as performance that fails to generalize relative to a reference benchmark and a fleet of reference models. We adopt this definition.
- DCR grades contamination at four levels and deflates accuracy with a fuzzy-inference factor.
Both output a model-level or benchmark-level scalar. Neither can say which number the model memorized, or whether failure reflects conflict or ignorance.
The method comes from the evidential-reasoning literature. Smets supplies the justification for non-additive belief with explicit ignorance. Fan et al. show that separating confusion from ignorance is practically useful: on CIFAR-100, retrieval of the true class among confused candidates reaches AUROC 90.0 versus 57.3 for softmax. Bi et al. is the precedent for buying tractability through restricted focal sets. Their "focal triplet" keeps the top two classes plus ignorance and finds that larger focal sets cost 48–78% more time for no accuracy gain.
Our restriction is different in kind. It is structural (intervals on an ordered frame), not truncation, so it discards nothing for single-marker thresholds. We fall back to triplet-style truncation only for the multi-marker case.
Research Program
Probe suite. We build a probe suite we call Threshold Probes. Each item is a short clinical vignette containing one biomarker value, an explicitly stated laboratory reference range or cutoff, and a question whose correct answer is determined by that stated reference. The suite has three tiers:
- Real markers with contested published cutoffs: NLR, AGR, GNRI, sFlt-1/PlGF and PlGF%.
- Markers drawn from LiveClin's lab tables and MIMIC-IV lab fields. These provide realism.
- Invented markers with invented reference ranges. Memorization is impossible here, which sets the reasoning ceiling.
For each item we sweep the value across a grid spanning all published cutoffs. Independently, we move the stated reference range, convert units (for example, g/L versus g/dL for albumin), and for gestational markers state different gestational ages. Each configuration is paraphrased several times; agreement across paraphrases gives the discount $\alpha$.
Evidential threshold estimator. Each answer becomes a discounted interval mass function. Sequential combination yields:
- belief and plausibility for every candidate threshold;
- the conflict mass $K$, the mass Dempster's rule would have placed on the empty set before normalization;
- the residual mass $m(\Theta)$.
We define two per-item readouts. Reference tracking is the plausibility that the implicit threshold lies within one grid step of the stated reference. Literature lock-in is the plausibility that it lies within one step of a published cutoff and stays there when the stated reference moves.
High lock-in with low conflict indicates memorization. High conflict indicates confusion between two thresholds, typically the stated one and the remembered one. High $m(\Theta)$ indicates ignorance. Item-level readouts aggregate to a model-by-biomarker map.
Multi-marker extension. For composite indices such as GNRI, or for vignettes with several labs, the joint frame explodes. Here we test whether keeping only the two most plausible joint threshold regions plus $\Theta$, a triplet in the sense of Bi et al., preserves the readouts. We validate against exact combination on small grids.
Baselines
Naive reference point. This is plain accuracy on the unperturbed vignettes, which is effectively what LiveClin and the MIMIC-IV LLM benchmark report. It is the "do nothing" point: if threshold memorization does not exist or does not matter, our readouts will add no information beyond this score. Reported reference numbers come from different tasks and serve only as context: o3 reaches 35.7% case accuracy on LiveClin, and zero-shot Llama 2-13B reaches AU-ROC 0.50 on MIMIC-IV mortality (Revisiting MIMIC-IV). There is no reported result on Lacuna for threshold-perturbed lab interpretation.
Strongest existing contamination detector. This is ConStat. We apply it with the original vignettes as the primary benchmark $D$ and the reference-shifted vignettes as $D_{ref}$, and fit its hardness-correction curve over the same fleet of open reference models we probe. It is the strongest competitor because it already detects non-generalizing performance without access to training data. It reports detecting 89% of syntax-contaminated and 98% of sample-contaminated models in controlled tests. For a fair comparison, both methods see the identical probe set, the identical reference-model fleet, and the same query budget. The comparison is run in the planted-contamination experiment below. We beat ConStat if we match its model-level detection rate and additionally recover which cutoff was planted. If ConStat alone matches us on detection, our contribution reduces to localization, and we will say so.
Fuzzy deflation. DCR is run with its four-level probes on the same items. It reports adjusted accuracy within 4% of uncontaminated baselines. We compare how well its DCR factor and our lock-in score each predict the true contamination status in the planted experiment.
Bayesian threshold model. This is the competitor most likely to tie us. It is a probit model with a latent threshold, a lapse rate, and a weak prior, fit to the same answers with the same probe budget.
Ablations. Four ablations isolate the key ingredients:
- drop discounting (all $\alpha = 1$);
- drop the conflict/ignorance split, reporting only pignistic-probability mass near each threshold;
- replace exact interval combination with triplet truncation, to measure what the truncation loses;
- remove the invented-marker control, to show what is lost without a reasoning ceiling.
Why not just…?
Why not just measure the accuracy drop when the stated reference range is shifted? This consistency score is cheap and will catch gross lock-in. It cannot distinguish a model that switches to a remembered cutoff from one that degrades to guessing: both show the same drop. That distinction changes the remedy. Retrieval of the right reference fixes memorization but not ignorance. In the planted experiment we will report whether the accuracy drop alone separates "planted-cutoff" from "noise-injected" models. If it does as well as our readouts, the evidential layer is unnecessary for detection and we will report that result.
Why not just fit a Bayesian latent-threshold model? This is a real threat. With many probes the posterior and our plausibility contours should agree. The question is the low-probe regime and the meaning of uncertainty. A Bayesian posterior with a weak prior spreads mass the same way whether answers conflict or are absent. Dempster's conflict mass and residual $m(\Theta)$ keep these apart, which is the property Fan et al. showed matters for retrieving confused candidates. We compare the two at probe budgets from 8 to 128 per item. If the Bayesian model localizes the planted threshold and separates the planted-versus-noise conditions equally well at every budget, we will adopt it and keep only the probe design and the cutoff-lock-in analysis as the contribution.
Why not just ask the model which cutoff it uses? Self-report is free and worth recording. However, stated and applied thresholds can diverge, much as models fluently complete memorized puzzles without computing them (Yax et al.). We will measure agreement between verbal self-report and the recovered implicit threshold. If agreement is high, self-report becomes the cheap baseline to beat.
Why not just use ConStat? ConStat answers "is this model inflated on this benchmark." It does not answer "which number is it inflated by." The latter determines whether a clinical deployment needs threshold standardization in the prompt or retrieval of local reference ranges.
Proposed Experiments
Planted contamination. This is the decisive check, using ground truth. We continue pretraining small open models on clinical-style documents in which a chosen real or invented biomarker is repeatedly paired with a planted cutoff. This mirrors ConStat's controlled contamination, but with the leaked content being a single number. Comparison arms are the clean model, a planted model, and a noise model trained on documents with randomly varying cutoffs. Success means:
- the recovered implicit threshold lands within one grid step of the planted value;
- lock-in is high and conflict low for the planted arm;
- conflict or $m(\Theta)$ rises for the noise arm;
- detection is at least matching ConStat on the same models.
Field study. We probe a fleet of open and API models on the real-marker tier and test whether recovered thresholds cluster at published values (3.24 for NLR, the 1.14–1.93 AGR range, 97 for GNRI) rather than at the stated reference. As a temporal control, we compare cutoffs from papers published before versus after each model's training cutoff. Lock-in should appear only for the former. The invented-marker tier gives each model's reasoning ceiling, so lock-in on real markers cannot be explained away as general inability to read a reference range.
Transfer to existing benchmarks. We test whether per-item lock-in on our probes predicts which LiveClin lab-interpretation questions and which MIMIC-IV lab-driven predictions a model gets right for the wrong reason. The test is whether those items fail when we rewrite their lab tables into a different unit or assay scale.
Tractability. We report wall-clock cost of exact interval combination against unrestricted power-set combination on small grids, where the latter is feasible. We also report the accuracy loss of triplet truncation on two- and three-marker joint thresholds.
Failure conditions. The proposal fails if planted cutoffs are not recoverable from black-box answers at a few hundred probes, or if the simple consistency drop or the Bayesian model matches our readouts in every condition. Both outcomes are informative, and the probe suite remains useful either way.
References
- Low Pretreatment Albumin-to-Globulin Ratio Predicts Poor Prognosis in Gastric Cancer: Insight From a Meta-Analysis
- Preoperative Neutrophil-to-Lymphocyte Ratio as a Predictive and Prognostic Factor for High-Grade Serous Ovarian Cancer
- Data classification using the Dempster-Shafer method
- Granular Association Rules for Multiple Taxonomies: A Mass Assignment Approach
- Comparison of three objective nutritional screening tools for identifying GLIM-defined malnutrition in patients with gastric cancer
- Maternal PlGF and sFlt-1 are Associated with Low Birth Weight and/or Small-for-Gestational Age Neonates in Pregnancy with or without Preeclampsia
- Plasma fibrin D-dimer and the risk of left atrial thrombus: A systematic review and meta-analysis
- LiveClin: A Live Clinical Benchmark without Leakage
- Revisiting the MIMIC-IV Benchmark: Experiments Using Language Models for Electronic Health Records
- ConStat: Performance-Based Contamination Detection in Large Language Models
- DCR: Quantifying Data Contamination in LLMs Evaluation
- Quantifying Beliefs by Belief Functions: An Axiomatic Justification
- Flexible Visual Recognition by Evidential Modeling of Confusion and Ignorance
- On Combining Multiple Classifiers Using an Evidential Approach
- Studying and improving reasoning in humans and machines