Single-Pass Decision Scores Enable Rapid Classification with Adjustable Thresholds in Medical Multimodal Large Language Models

Patrick Wienholt, Frederik Hauke, Christiane Kuhl, Jakob Nikolas Kather, Sven Nebelung, Daniel TruhnarXiv 2026 · arXiv

TL;DR

Generative multimodal large language models (MLLMs) typically provide binary "yes/no" answers using a fixed, implicit decision threshold that is often poorly suited for clinical tasks. This paper introduces a simple post-hoc method to extract continuous decision scores from a single forward pass, enabling clinicians to tune sensitivity and specificity without retraining the model.

What Problem Does This Paper Address?

In medicine, the "right" answer depends on the context. A screening tool for breast cancer should prioritize sensitivity (don't miss any cases), while a tool deciding on a risky surgery should prioritize specificity (don't operate on healthy people).

Standard MLLMs, however, are generative. When asked "Does this X-ray show pneumonia?", they sample the most likely next token—usually "yes" or "no." This effectively hard-codes a decision threshold at 0.5. If the model is slightly biased toward "no," it might miss critical findings entirely at this default setting. Furthermore, the raw probability of a "yes" token is often "diluted" by the thousands of other unrelated tokens in the model's vocabulary, making it an unreliable confidence score.

Methods

The authors propose a Single-Pass Decision Score approach. Instead of letting the model generate a full text response, they look at the probability distribution of the very first token.

  1. Extraction: They extract the raw probabilities for the tokens "yes" and "no" ($P_{raw}(\text{yes})$ and $P_{raw}(\text{no})$).
  2. Normalization: They renormalize these two values so they sum to 1: $$P_{norm}(\text{yes}) = \frac{P_{raw}(\text{yes})}{P_{raw}(\text{yes}) + P_{raw}(\text{no})}$$ This isolates the model's binary choice from the "noise" of the rest of the vocabulary.

  3. Thresholding: This normalized score is used as a continuous variable. By sweeping this score from 0 to 1, they can generate Receiver Operating Characteristic (ROC) curves and select an optimal operating point based on a validation set.

Fig. 1: Workflow for threshold shifting in LLM-based binary classification. (a) An image-conditioned prompt is provided to the LLM. (b) From the first decoding step, the model assigns probabilities to all vocabulary tokens, including the verbalizers yes and no. (c) We normalize the probabilities over {yes, no} and vary the decision threshold, which has an implicit default of 0.5. (d) Sweeping the threshold yields different operating points on the ROC curve, trading true positive rate (TPR) against false positive rate (FPR). The optimal threshold was determined on a validation set disjoint from the test set.

The researchers compared this "Normalized" approach against using "Raw" probabilities and "Text-elicited" probabilities (where the model is prompted to explicitly write out a numerical probability, e.g., "0.85").

Datasets

The method was evaluated across six diverse medical imaging datasets:

  • Chest Radiography: CheXpert and MIMIC-CXR-JPG.
  • Dermatology: HAM10000 and DERM12345.
  • Ophthalmology: G1020 (glaucoma detection).
  • Histopathology: NCT-CRC-HE (colorectal cancer tissue classification).
Fig. 2: Overview of the experimental setup. We evaluated the threshold-shifting method on six medical imaging datasets spanning chest radiography, dermatology, ophthalmology, and histopathology. Notably, CheXpert was explicitly excluded from the training data of the MedGemma models [18], making it well suited for external validation. Each dataset was split into a validation set, used exclusively to select the decision threshold, and a test set, used only for the final performance evaluation. We compared four models, namely Gemma-3 and MedGemma each in a 4B and a 27B variant, using three scoring methods: the normalized yes/no token probability, the raw yes token probability, and a text-elicited probability generated directly as out- put. The threshold optimized on the validation set was compared against the model’s implicit default threshold of 0.5 on the test set.

Key Insights and Ideas

  • The Default is Often Broken: The most striking finding was how poorly the default 0.5 threshold performed. For identifying "normal" chest X-rays (the "No Finding" class), MedGemma-27B at default settings labeled every single image as abnormal (0% sensitivity). By simply shifting the threshold to 0.005, sensitivity jumped to 94.5%.
  • Normalization as Denoising: Normalizing the yes/no probabilities consistently outperformed asking the model to "think" of a probability and write it down. This suggests that MLLMs have better internal discriminative signals than they are able to express through text generation.
  • Domain-Specific Fine-Tuning Matters (Mostly): MedGemma (fine-tuned for medicine) crushed the general-purpose Gemma-3 on X-ray tasks. However, surprisingly, the general models actually performed better on some dermatology datasets, highlighting that medical fine-tuning isn't a universal "level up."
  • Efficiency: Because the method only requires the first decoding step, it is incredibly fast—requiring only ~50-80ms per decision.

Key Results

The normalization approach achieved the highest Area Under the ROC Curve (AUROC) on four of the six datasets.

  • CheXpert: The normalized score reached an AUROC of 0.819, significantly higher than raw probabilities (0.777) or text-elicited probabilities (0.743).
  • Class-Specific Performance: As shown in Figure 3, the benefit of normalization is widespread across different medical findings, though some classes (like "Pleural Other") remain challenging for current models.
Fig. 3: Class-specific AUROC values for the CheXpert dataset using MedGemma- 27B. The three bars for each class represent normalized token probabilities, raw token probabilities (“Unnormalized” in the legend), and text-elicited probability estimates (“Text”). Error bars indicate 95% confidence intervals. The “average” bars show the mean across classes.

The impact of threshold shifting is best visualized through ROC curves. In Figure 4, we see that the "Optimal Threshold" (orange dot) is often far from the "Default 0.5" (blue dot), drastically changing the clinical utility of the model.

Fig. 4: ROC curves showing the difference between default and optimal thresholds.

Limitations and Future Work

  • Logit Access: This method requires access to token-level probabilities (logits). Many commercial MLLMs (like GPT-4o or Gemini) accessed via API do not always provide these, or provide them in a limited capacity.
  • Binary Limitation: The current study focuses on binary "yes/no" questions. Extending this to multi-class problems (e.g., "Which of these 5 diseases is present?") would require more complex normalization across a larger set of verbalizer tokens.
  • Prompt Sensitivity: Like all LLM tasks, the results are sensitive to how the prompt is phrased. Future work could investigate how to make these decision scores more robust to prompt variations.
  • Reasoning vs. Intuition: The authors note that while this "System One" (fast, intuitive) approach works well, it remains to be seen how "Chain-of-Thought" reasoning might bias or improve these underlying probabilities.

Summary updated

Abstract

Multimodal large language models (MLLMs) answer binary clinical questions with a yes or no token, leaving the decision threshold implicit and fixed. We derive a continuous score from a single forward pass by renormalizing the probabilities of the yes and no tokens, allowing the threshold to be adapted to clinical needs without retraining. We evaluated this approach with MedGemma and Gemma-3 (4B and 27B) on six medical imaging datasets. The default threshold was frequently misaligned with the task: for identifying normal chest radiographs, MedGemma-27B labeled every image as abnormal, whereas a validation-selected threshold yielded a sensitivity of 0.945 and a specificity of 0.696. Normalization achieved the highest AUROC on four of six datasets (CheXpert: 0.819 vs. 0.777 for raw and 0.743 for text-elicited probabilities), and a single decision required 53-79 ms on one GPU. This simple post-hoc method turns medical MLLMs into fast classifiers with adjustable operating points. Code: https://github.com/TruhnLab/llm-threshold-calibration
BibTeX
@misc{wienholt2026single,
  author = {Patrick Wienholt and Frederik Hauke and Christiane Kuhl and Jakob Nikolas Kather and Sven Nebelung and Daniel Truhn},
  title = {{Single-Pass Decision Scores Enable Rapid Classification with Adjustable Thresholds in Medical Multimodal Large Language Models}},
  year = {2026},
  eprint = {2610.23988},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.23988},
}
Figure