Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs

Yunpeng Xiao, Carl Yang, Mark Mai, Xiao Hu, Kai ShuarXiv 2025 · arXiv · OpenReview

Abstract

Large language models (LLMs) show promise for clinical use. They are often evaluated using datasets such as MedQA. However, Many medical datasets, such as MedQA, rely on simplified Question-Answering (Q\A) that underrepresents real-world clinical decision-making. Based on this, we propose a unifying paradigm that characterizes clinical decision-making tasks along two dimensions: Clinical Backgrounds and Clinical Questions. As the background and questions approach the real clinical environment, the difficulty increases. We summarize the settings of existing datasets and benchmarks along two dimensions. Then we review methods to address clinical decision-making, including training-time and test-time techniques, and summarize when they help. Next, we extend evaluation beyond accuracy to include efficiency, explainability. Finally, we highlight open challenges. Our paradigm clarifies assumptions, standardizes comparisons, and guides the development of clinically meaningful LLMs.

Introduction: The Gap Between Exams and Practice

Current evaluation of medical AI relies heavily on "sanitized" datasets like MedQA, which consist of clean, multiple-choice questions derived from medical board exams. While LLMs have achieved expert-level performance on these benchmarks, a significant gap remains between passing a test and practicing medicine. In real-world clinical settings, information is rarely presented in a neat paragraph. Clinicians must navigate incomplete data, sift through noisy Electronic Health Records (EHRs), and make open-ended decisions that involve ethical nuances. This paper proposes a new framework to bridge this gap by categorizing clinical decision-making (CDM) tasks across two critical dimensions.

A New Paradigm for Clinical Decision-Making

The authors propose a 4x4 matrix to classify CDM tasks based on the complexity of the input and the nature of the output.

1. Clinical Backgrounds (The Input)

The "Background" dimension represents the information available to the model. It ranges from:

  • Precise Information: Concise, textbook-style summaries.
  • Rich Information: Large-scale, multimodal data including EHR notes, ECGs, and medical images, often containing redundant or noisy information.
  • Incomplete Information: Scenarios where the model must interactively gather data from a patient or system to reach a conclusion.
The three levels of Clinical Backgrounds: Precise (textbook), Rich (multimodal EHR), and Incomplete (interactive dialogue).

2. Clinical Questions (The Output)

The "Question" dimension categorizes the required response, ranging from simple binary choices (Yes/No) and Multiple Choice Questions (MCQs) to Short-answer (e.g., specific dosages) and complex Open-ended questions (e.g., treatment planning).

Mapping the Benchmark Landscape

By applying this paradigm, the authors conducted a meta-analysis of existing medical datasets. Most traditional benchmarks (e.g., MMLU, MedQA) cluster in the "Precise Background + Multiple Choice" quadrant. Newer datasets are beginning to explore more challenging areas, such as AgentClinic for incomplete information and HealthBench for open-ended ethical reasoning.

A meta-analysis mapping existing medical benchmarks across the dimensions of Clinical Backgrounds and Clinical Questions.

Technical Strategies for Real-World CDM

To improve LLM performance in these complex settings, the paper reviews several technical approaches:

  • Training-time Techniques: Beyond standard Supervised Fine-Tuning (SFT), Reinforcement Learning (RL) methods like DPO and GRPO are used to align models with clinical reasoning paths.
  • Test-time Techniques: Strategies like Chain-of-Thought (CoT) reasoning and Retrieval-Augmented Generation (RAG) help models handle long contexts. Multi-agent systems, where one LLM simulates a doctor and another a patient, are particularly effective for "Incomplete Information" tasks.

Key Insights and Challenges

The analysis reveals several critical findings:

  • The Complexity Penalty: LLM accuracy drops significantly when moving from "Precise" to "Incomplete" information backgrounds. Even state-of-the-art models like GPT-4o struggle with active information gathering.
  • The Redundancy Problem: In "Rich" backgrounds, models often suffer from being "lost in the middle" of long EHR contexts, where irrelevant data degrades reasoning.
  • Beyond Accuracy: Evaluation must evolve to include Efficiency (reaching a diagnosis in fewer steps) and Explainability (the quality of the reasoning path).

Future Directions

The path toward clinically useful LLMs requires addressing several "frontier" challenges. These include moving from synthetic to real-world de-identified clinical data, improving the realism of "Patient Agents," and developing robust "label-free" metrics for evaluating open-ended clinical decisions where no single "ground truth" exists.

Summary updated

Abstract

Large language models (LLMs) show promise for clinical use. They are often evaluated using datasets such as MedQA. However, Many medical datasets, such as MedQA, rely on simplified Question-Answering (Q\A) that underrepresents real-world clinical decision-making. Based on this, we propose a unifying paradigm that characterizes clinical decision-making tasks along two dimensions: Clinical Backgrounds and Clinical Questions. As the background and questions approach the real clinical environment, the difficulty increases. We summarize the settings of existing datasets and benchmarks along two dimensions. Then we review methods to address clinical decision-making, including training-time and test-time techniques, and summarize when they help. Next, we extend evaluation beyond accuracy to include efficiency, explainability. Finally, we highlight open challenges. Our paradigm clarifies assumptions, standardizes comparisons, and guides the development of clinically meaningful LLMs.
BibTeX
@misc{xiao2025beyond,
  author = {Yunpeng Xiao and Carl Yang and Mark Mai and Xiao Hu and Kai Shu},
  title = {{Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs}},
  year = {2025},
  eprint = {2510.20001},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.20001},
}
Figure