Abstract
Generating accurate radiology reports requires precise visual interpretation and structured medical reasoning. We introduce Thought Graph Traversal (TGT), a framework that enhances Vision-Language Large Models (VLLMs) by scaling test-time compute through structured diagnostic paths. By guiding models through a hierarchical graph of organ-specific questions and verification steps, TGT achieves significant performance gains on IU X-Ray and MIMIC-CXR benchmarks without further model training.
Introduction
Radiology report generation is a high-stakes task where modern VLLMs often struggle with structural consistency and logical depth. Models frequently exhibit "shallow reasoning" or hallucinations, optimizing for linguistic probability rather than diagnostic accuracy. While fine-tuning is a common remedy, it is computationally expensive. This work explores whether structured "thinking" during inference—test-time scaling—can bridge the gap between general-purpose VLLMs and expert-level radiological reporting.
Methodology: Thought Graph Traversal (TGT)
The core of our approach is the Thought Graph Traversal (TGT) framework. Instead of a single-pass generation, the model follows a structured hierarchy to build the report.
The process involves:
- Thought Graph Construction: A two-level map where the first level identifies organs (e.g., Heart, Lungs) and the second level defines specific diagnostic questions (e.g., "Is the heart size normal?").
- Sequential Traversal & Verification: The model iterates through these nodes, performing a verification step after each query to ensure findings align with the visual evidence.
- Reasoning Budget Scaling: By adjusting the depth and length of the reasoning chain, we control the "compute" allocated to the task.
- In-Context Learning (ICL): Expert examples are retrieved from a database to guide the description of specific findings.
Experimental Setup and Scaling Insights
We evaluated TGT on the IU X-Ray and MIMIC-CXR datasets. A key finding is the existence of a "saturation point" for reasoning.
As shown in the performance plots, increasing the "reasoning budget" (measured in tokens) leads to a steady improvement in metrics like BLEU-4 and ROUGE-L, eventually plateauing around 450 tokens. This suggests that while "thinking harder" helps, there is an optimal compute allocation for radiology tasks. Furthermore, we found that the order of organ analysis matters; models that follow a standard radiological sequence produce more coherent reports.
Results
TGT consistently outperforms zero-shot and standard few-shot prompting across models like GPT-4o and Qwen2.5-VL. On the MIMIC-CXR dataset, GPT-4o’s BLEU-4 score improved nearly tenfold, from 1.03 (Zero-Shot) to 9.84 (TGT). Compared to other reasoning strategies like Rephrase and Respond (RaR), TGT achieved significantly higher ROUGE-L scores (37.33 vs 17.74 on IU X-Ray), demonstrating the value of domain-specific structural guidance.
Discussion and Conclusion
While TGT significantly improves accuracy and clinical coherence, it introduces higher inference latency due to its multi-step nature. Future work will investigate more complex graph structures and multi-agent systems to further refine the diagnostic process. In conclusion, TGT demonstrates that structured test-time scaling is a powerful, training-free method for elevating VLLM performance in specialized medical domains.
Paper Citation: Yao, Y., Wen, Z., Tong, Y., et al. (2025). Simple Radiology VLLM Test-time Scaling with Thought Graph Traversal. arXiv:2506.11989v2 [cs.CV].
Summary updated