TL;DR
DiagTrace is a new interactive benchmark that shifts clinical LLM evaluation from static "final diagnosis" guessing to the active process of information acquisition. By providing models with a final diagnosis and asking them to reconstruct the diagnostic path through a turn-based dialogue, it reveals how well AI can prioritize clinical evidence, follow logical ordering, and manage a fixed "turn budget."
What Problem Does This Paper Address?
Most current medical LLM benchmarks are "static": they provide a complete patient history and test results upfront, then ask for a diagnosis. While this tests medical knowledge, it ignores the reality of clinical practice, where diagnosis is an incremental information-acquisition process. A doctor must decide what to ask, what to examine, and which tests to order—often under constraints of time and cost.
Existing interactive benchmarks often conflate the ability to find information with the ability to infer a diagnosis. DiagTrace isolates the "seeking" behavior. It asks: "Given the correct diagnosis, can a model actively reconstruct the diagnostic trajectory?" This forces the model to prioritize high-yield requests and balance information coverage against interaction costs.
Methods
The authors developed a three-stage pipeline to evaluate "Interactive Diagnostic Reconstruction":
- Reference Resource Construction: They converted PubMed Central (PMC) case reports into structured resources containing an initial presentation, an "event bank" of findings, a clinical course, and diagnostic references.
- Interactive Protocol: Two agents are initialized. The Clinician Agent (the LLM under test) receives the initial presentation and the final diagnosis. It must then issue focused requests (e.g., "order a chest CT") to a Report Agent. The Report Agent, grounded in the source case, returns the relevant finding if it exists in the event bank.
- Trajectory-Level Evaluation: Instead of just checking the final output, the authors measure the entire path. They map requests to a shared taxonomy of 10 clinical categories (e.g., Imaging, Pathology, Lab tests) and use metrics like:
- Tversky Alignment: Measuring category-level coverage while penalizing unnecessary or redundant turns.
- Longest Common Subsequence (LCS) Recall: Checking if the model recovers the source clinical course in the correct relative order.
- Pairwise Order Accuracy: Ensuring the sequence of requests makes clinical sense.
Datasets
The benchmark, DiagTrace, consists of 510 cases spanning eight clinical specialties (Dermatology, Oncology, Radiology, etc.).
- Source: PubMed Central (PMC) case reports.
- Scale: 5,911 unique source events.
- Diversity: Each case contains an average of ~11.6 source events, ranging from physical exams to complex molecular pathology.
Key Insights and Ideas
- Adaptive Seeking Matters: The authors compared LLMs against a "fixed-checklist" baseline (a model that asks the same questions in the same order every time). All LLMs outperformed the fixed baseline, proving they actually use the provided diagnosis to adaptively guide their search for evidence.
- The Coverage-Efficiency Trade-off: Models that found the most information (high recall) weren't always the most efficient. For example, GPT-4.1 mini used the fewest turns on average but had lower overall coverage compared to Kimi K2.5.
- Category Bias: Models are remarkably consistent at requesting "hard" data like Lab tests and Imaging but frequently overlook "softer" but vital information like physical examination findings or treatment response.
Key Results
The study evaluated four major models: GPT-4.1 mini, Claude Haiku 4.5, Gemini 2.5 Flash, and Kimi K2.5.
- Top Performer (Alignment): Kimi K2.5 achieved the highest Tversky alignment (0.399) and LCS recall (0.561), suggesting it is best at reconstructing the breadth of the diagnostic path.
- Top Performer (Ordering): Claude Haiku 4.5 excelled in Pairwise order accuracy (0.690), meaning its sequence of requests most closely mirrored the logical flow of a real clinical case.
- Efficiency: GPT-4.1 mini was the most concise, using an average of only 9.93 turns (out of a 15-turn budget) to complete its reconstruction.
- Quality Audit: Human and LLM-based audits confirmed that the extracted events were highly faithful to the original source articles (>98% faithfulness in LLM audits).
Limitations and Future Work
- Narrative vs. Chronology: Case reports are written as narratives. The "order" of events in a paper might reflect a storytelling choice by the authors rather than the strict chronological order in which tests were performed in the hospital.
- Selection Bias: Case reports typically highlight unusual or "perfect" diagnostic paths; they may not reflect the messy, redundant workflows of routine clinical care.
- LLM-as-Judge: While the authors used multiple models for auditing, the benchmark construction still relies heavily on LLM assistance, which could inherit systemic biases.
Future work could involve expanding to multi-modal evidence (interpreting raw images rather than text captions) and evaluating models in "blind" settings where the final diagnosis is not provided, testing both acquisition and inference simultaneously.
Summary updated