Simple Radiology VLLM Test-time Scaling with Thought Graph Traversal

Yue Yao, Zelin Wen, Yan Tong, Xinyu Tian, Xuqing Li, Xiao Ma, Dongliang Xu, Tom Gedeon

Abstract

Generating accurate radiology reports requires precise visual interpretation and structured medical reasoning. We introduce Thought Graph Traversal (TGT), a framework that enhances Vision-Language Large Models (VLLMs) by scaling test-time compute through structured diagnostic paths. By guiding models through a hierarchical graph of organ-specific questions and verification steps, TGT achieves significant performance gains on IU X-Ray and MIMIC-CXR benchmarks without further model training.

Introduction

Radiology report generation is a high-stakes task where modern VLLMs often struggle with structural consistency and logical depth. Models frequently exhibit "shallow reasoning" or hallucinations, optimizing for linguistic probability rather than diagnostic accuracy. While fine-tuning is a common remedy, it is computationally expensive. This work explores whether structured "thinking" during inference—test-time scaling—can bridge the gap between general-purpose VLLMs and expert-level radiological reporting.

Methodology: Thought Graph Traversal (TGT)

The core of our approach is the Thought Graph Traversal (TGT) framework. Instead of a single-pass generation, the model follows a structured hierarchy to build the report.

The TGT workflow involves a hierarchical traversal of organs and diagnostic aspects, incorporating verification steps and in-context learning.

The process involves:

  1. Thought Graph Construction: A two-level map where the first level identifies organs (e.g., Heart, Lungs) and the second level defines specific diagnostic questions (e.g., "Is the heart size normal?").
  2. Sequential Traversal & Verification: The model iterates through these nodes, performing a verification step after each query to ensure findings align with the visual evidence.
  3. Reasoning Budget Scaling: By adjusting the depth and length of the reasoning chain, we control the "compute" allocated to the task.
  4. In-Context Learning (ICL): Expert examples are retrieved from a database to guide the description of specific findings.

Experimental Setup and Scaling Insights

We evaluated TGT on the IU X-Ray and MIMIC-CXR datasets. A key finding is the existence of a "saturation point" for reasoning.

Performance metrics (BLEU, METEOR, ROUGE_L) across IU X-Ray and MIMIC-CXR datasets show a clear correlation with the number of reasoning tokens, plateauing around 450 tokens.

As shown in the performance plots, increasing the "reasoning budget" (measured in tokens) leads to a steady improvement in metrics like BLEU-4 and ROUGE-L, eventually plateauing around 450 tokens. This suggests that while "thinking harder" helps, there is an optimal compute allocation for radiology tasks. Furthermore, we found that the order of organ analysis matters; models that follow a standard radiological sequence produce more coherent reports.

Results

TGT consistently outperforms zero-shot and standard few-shot prompting across models like GPT-4o and Qwen2.5-VL. On the MIMIC-CXR dataset, GPT-4o’s BLEU-4 score improved nearly tenfold, from 1.03 (Zero-Shot) to 9.84 (TGT). Compared to other reasoning strategies like Rephrase and Respond (RaR), TGT achieved significantly higher ROUGE-L scores (37.33 vs 17.74 on IU X-Ray), demonstrating the value of domain-specific structural guidance.

Discussion and Conclusion

While TGT significantly improves accuracy and clinical coherence, it introduces higher inference latency due to its multi-step nature. Future work will investigate more complex graph structures and multi-agent systems to further refine the diagnostic process. In conclusion, TGT demonstrates that structured test-time scaling is a powerful, training-free method for elevating VLLM performance in specialized medical domains.

Paper Citation: Yao, Y., Wen, Z., Tong, Y., et al. (2025). Simple Radiology VLLM Test-time Scaling with Thought Graph Traversal. arXiv:2506.11989v2 [cs.CV].

Summary updated

Abstract

Test-time scaling offers a promising way to improve the reasoning performance of vision-language large models (VLLMs) without additional training. In this paper, we explore a simple but effective approach for applying test-time scaling to chest X-ray report generation. Specifically, we introduce a lightweight Thought Graph Traversal (TGT) framework that guides the model to reason through organ-specific findings in a medically coherent order. This framework integrates structured medical priors into the prompt, enabling deeper and more logical analysis with no changes to the underlying model. To further enhance reasoning depth, we apply a reasoning budget forcing strategy that adjusts the model's inference depth at test time by dynamically extending its generation process. This simple yet powerful combination allows a frozen radiology VLLM to self-correct and generate more accurate, consistent chest X-ray reports. Our method outperforms baseline prompting approaches on standard benchmarks, and also reveals dataset biases through traceable reasoning paths. Code and prompts are open-sourced for reproducibility at https://github.com/glerium/Thought-Graph-Traversal
BibTeX
@article{yao2026simple,
  author = {Yue Yao and Zelin Wen and Yan Tong and Xinyu Tian and Xuqing Li and Xiao Ma and Dongliang Xu and Tom Gedeon},
  title = {{Simple Radiology VLLM Test-time Scaling with Thought Graph Traversal}},
  journal = {Pattern Recognition},
  year = {2026},
  doi = {10.1016/j.patcog.2026.113639},
}
Figure