From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations

Benlu Wang, Iris Xia, Yifan Zhang, Junda Wang, Feiyun Ouyang, Shuo Han, Arman Cohan, Hong Yu, Zonghai Yao

Introduction

Large Language Models (LLMs) are increasingly being integrated into clinical decision support systems. However, in the medical domain, the difference between a "close enough" answer and a precise calculation can be a matter of patient safety. Traditional evaluations of LLMs in medical contexts often focus solely on the final numeric output, allowing for a margin of error that may mask significant underlying reasoning failures. This paper addresses the "black-box" nature of these evaluations by introducing a granular, step-by-step diagnostic framework and MedRaC, an agentic pipeline designed to ensure both accuracy and transparency in medical calculations.

The Problem: The Illusion of Performance

Current benchmarks, such as the original MedCalc-Bench, typically evaluate models based on their final answers, often permitting a ±5% margin of error. The authors argue that this approach is insufficient for clinical safety. An LLM might produce a correct-looking number through "hallucinated arithmetic" or by applying an incorrect formula that happens to yield a similar result in a specific case.

Such systematic reasoning failures—including the misidentification of patient attributes or the failure to apply critical demographic multipliers (e.g., age or sex factors)—are often hidden behind a "correct" score. When the researchers applied a more rigorous step-by-step evaluation, the performance of top-tier models like GPT-4o plummeted from 62.7% to 43.6%. This suggests that nearly a third of "correct" answers in previous benchmarks were actually the result of flawed reasoning or lucky guesses, creating a dangerous "illusion of performance."

A Granular Diagnostic Framework

To move from simple "scores" to verifiable "steps," the researchers proposed a three-part framework to diagnose and rectify calculation errors:

  1. Step-wise Evaluation Pipeline: The calculation process is decomposed into four individually validated stages: Formula Selection, Value Extraction (from clinical notes), Mathematical Calculation, and Final Answer Formatting.
  2. LLM-as-Judge Error Attribution: An automated judge was developed to categorize failures into a specific taxonomy, such as Formula Misselection, Unit Conversion Errors, and Demographic Adjustment Failures. This automated system achieved a 90.2% agreement rate with expert clinicians, proving its reliability for scalable diagnostics.
  3. MedRaC (Medical Retrieval and Code): A modular, training-free agentic pipeline designed to solve the identified issues.

MedRaC: Bridging the Gap with RAG and Code

The MedRaC pipeline addresses the two primary failure points of LLMs: formula knowledge and arithmetic precision.

  • Formula-RAG: Most LLMs do not reliably memorize the precise coefficients and structures of complex medical formulas. MedRaC uses a Retrieval-Augmented Generation (RAG) approach to fetch the exact, canonical formula from a trusted database (such as MDCalc) and injects it directly into the model's prompt.
  • Code Execution: To eliminate the "hallucinated sums" common in multi-step arithmetic, MedRaC instructs the model to write Python code to perform the calculation. By delegating the math to a deterministic interpreter, the system ensures perfect arithmetic precision, allowing the LLM to focus on the "human" task of extracting variables from messy clinical notes.

Experimental Setup and Results

The researchers re-audited the MedCalc-Bench dataset, removing 108 faulty or obsolete records to create a high-quality set of 940 valid clinical cases covering 55 different medical calculators. They also tested the system against a formula library scaled to 785 entries to ensure robustness.

Key Findings

  • Significant Accuracy Gains: MedRaC improved calculation accuracy across various model sizes. Most notably, smaller models like Phi-4-mini saw their accuracy jump from a baseline of 2.16% to 68.39%.
  • Retrieval as a Bottleneck: Ablation studies confirmed that Formula-RAG is critical; without it, accuracy dropped from approximately 65% to 26%, highlighting that even advanced models struggle to recall specific medical formulas from their internal weights.
  • Error Reduction: In a case study using Llama-3.1-8B, MedRaC reduced formula-related errors by 77.5% and arithmetic errors by 82.6% compared to standard Chain-of-Thought (CoT) prompting.

Discussion and Limitations

While MedRaC significantly advances the state of the art, challenges remain. The system still struggles with "implicit context"—cases where clinical variables are not explicitly stated in the notes and require nuanced medical background knowledge to infer. Furthermore, the current study focused on English-language curated notes; future work must address the performance of these systems on multilingual data and the "noisy" reality of Electronic Health Records (EHRs).

Conclusion

By shifting the focus from final scores to verifiable steps, this research provides a roadmap for safer clinical AI. The MedRaC pipeline demonstrates that combining specialized retrieval with code execution can transform LLMs from unreliable calculators into precise clinical tools. However, the authors emphasize that in high-stakes environments, these systems should serve as support for clinicians rather than operating with full autonomy, eventually integrating real-time human oversight.

Summary updated

BibTeX
@inproceedings{wang2025from,
  author = {Benlu Wang and Iris Xia and Yifan Zhang and Junda Wang and Feiyun Ouyang and Shuo Han and Arman Cohan and Hong Yu and Zonghai Yao},
  title = {{From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=3g7bnBX7DS},
}
Figure