Can decision models extract clinical data from cancer pathology reports? An evaluation of accuracy, calibration, and failure modes

Indranil Mallick, Paromita RoyarXiv 2026 · arXiv

This paper explores a new frontier in clinical NLP: Decision Models. Unlike standard Large Language Models (LLMs) that generate text, decision models are designed to choose from a fixed list of options and provide a probability score for each. The authors evaluate whether these specialized models can accurately and efficiently extract structured data from complex, narrative cancer pathology reports.

TL;DR

Decision models can extract clinical data from pathology reports as accurately as generative LLMs but are up to four times faster and run efficiently on local hardware. While they provide well-calibrated confidence scores that allow for automated "cascading" to stronger models, they struggle with numeric values and require careful prompt engineering to avoid performance drops when asking multiple questions.

What Problem Does This Paper Address?

Clinical data extraction is a bottleneck in oncology. Pathology reports contain the "ground truth" for cancer staging and treatment planning, but they are often written as free-text narratives. Manually turning these into structured database entries is slow, expensive, and prone to human error.

While generative LLMs (like GPT-4 or Llama) are good at this, they have two major drawbacks in a hospital setting:

  1. Parsing overhead: They output text that must be cleaned and validated against a schema.
  2. Poor calibration: They are notoriously bad at knowing when they are wrong. They don't provide a reliable "confidence score" out of the box.

Decision models aim to solve this by treating extraction as a classification task rather than a generation task, potentially offering better speed, local deployment (for data privacy), and built-in uncertainty quantification.

Methods

The researchers compared two open-weights decision models—Nimble 9B and Tev1 4B—against two generative benchmarks: Qwen3.5 9B (local) and GLM-5.3-flash (cloud API).

Key technical aspects included:

  • Task: Extracting 25 distinct clinical fields (e.g., tumor grade, margin status, lymph node involvement).
  • Hardware: Local models were run on an Intel Arc GPU (32 GB RAM), emphasizing the feasibility of "on-premise" deployment.
  • Calibration Analysis: The authors measured how well the models' predicted probabilities matched their actual accuracy using Expected Calibration Error (ECE) and Brier scores.
  • The Cascade Strategy: They simulated a workflow where any answer with a low probability score was automatically "escalated" to a more powerful cloud model or a human reviewer.

Datasets

The study used a specialized, real-world dataset:

  • 1,113 oral cavity cancer pathology reports from a tertiary care center in India.
  • Reference Standard: A prospectively maintained clinical database, verified by expert clinicians and pathologists.
  • Fields: 16 categorical fields and 6 numeric fields (binned into categories for the decision models).

Key Insights and Ideas

  • The Speed-Accuracy Sweet Spot: Tev1 4B matched the accuracy of the much larger Qwen3.5 9B generative model but was 4x faster (15 seconds vs. 61 seconds per report). This suggests that for structured extraction, "choosing" is more efficient than "writing."
  • The "Batching" Penalty: A surprising finding was that Nimble 9B’s accuracy tanked when asked 25 questions in a single request but improved significantly when asked only two. This implies that decision models may suffer from "internal competition" or attention interference when the output space becomes too crowded.
  • Probabilities are Actually Useful: Unlike generative models, the decision models' probability outputs were well-calibrated (ECE ~0.02). This means if the model says it's 95% sure, it's actually right about 95% of the time.
  • Numeric Blind Spots: Decision models struggled with binned numeric values (e.g., "is the tumor >20mm?"). Generative models performed better here because they simply "copy-pasted" the number from the text, leaving the logic to a post-processing script.

Key Results

  • Overall Accuracy: Tev1 4B achieved a Mean Field Accuracy (MFA) of 0.917, identical to the generative Qwen3.5 9B. The cloud-based GLM-5.3-flash remained the leader at 0.986.
  • Efficiency: Tev1 used significantly fewer tokens and less time than generative counterparts.
  • The Power of the Cascade: By escalating only the bottom 38% of Tev1’s least-confident answers to the cloud-based GLM model, the system's overall accuracy jumped to 0.982. This provides a blueprint for high-accuracy systems that keep most data local and only use expensive/cloud resources when necessary.

Limitations and Future Work

  • Systematic Errors: While probabilities caught random "guesses," they didn't flag systematic errors (e.g., Tev1 consistently failing to recognize "Bone not resected").
  • Single-Center Study: The reports were from one hospital in one language; performance might vary with different reporting styles.
  • Numeric Extraction: The authors suggest that future pipelines should be hybrid: use decision models for categorical "presence/absence" findings and generative models or regex for precise numeric measurements.
  • Prompt Sensitivity: The study highlighted that "request design" (how many questions you ask at once) is just as important as the model weights themselves.

Summary updated

Abstract

Decision models are a new class of language model that, instead of generating text, choose one option per question from a fixed list and return a probability for every option. They have not yet been evaluated on clinical text. We evaluated two open-weights decision models, Nimble 9B and Tev1 4B, for extracting 25 fields from 1,113 oral cavity cancer pathology reports from a single centre, against a prospectively maintained clinical database. We compared them with two generative models: Qwen3.5 9B, run locally, and GLM-5.3-flash, run through a cloud API. Tev1 reached a mean field accuracy of 0.917 (95% CI 0.913 to 0.920), the same as Qwen3.5 9B (0.918; p = 0.26), and was four times faster (15 s against 61 s per report). Nimble reached 0.862 and GLM-5.3-flash 0.986. Both decision models matched GLM-5.3-flash on simple dichotomous categories. They were weak on measurements binned into categories, on fields whose correct answer was not reported, and on some complex decisions (e.g. involved margins). We found that the main cause of the inferior performance of Nimble's came from asking 25 questions in one request: with two questions per request, its accuracy improved. Pooled over fields, the returned probabilities were well calibrated (expected calibration error 0.022 and 0.026), but they did not flag systematic errors within single fields. A simulated strategy of sending answers with a probability below 0.94 to GLM-5.3-flash raised Tev1's accuracy to 0.982, with 38% of answers escalated. Decision models can extract explicitly stated findings from pathology reports quickly on local hardware and return useful probabilities. They should be validated in the configuration in which they will be used, restricted to the fields in which they are reliable, and combined with other methods for numeric values.
BibTeX
@misc{mallick2026can,
  author = {Indranil Mallick and Paromita Roy},
  title = {{Can decision models extract clinical data from cancer pathology reports? An evaluation of accuracy, calibration, and failure modes}},
  year = {2026},
  eprint = {2611.20129},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20129},
}
Figure