CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Training

Jiahe Qian, Yuhao Shen, Zhangtianyi Chen, Juexiao Zhou, Peisong WangarXiv 2025 · DOI · arXiv

Introduction

Medical Visual Question Answering (VQA) systems—designed to assist clinicians by interpreting X-rays, CT scans, and pathology slides—face a significant "reliability gap." When deployed in real-world clinical environments, these models often encounter domain shift, where differences in imaging equipment, protocols, or patient demographics lead to performance degradation. Under these conditions, even state-of-the-art models are prone to "hallucinations" or attending to spurious background regions rather than the relevant pathology.

Traditional fine-tuning is often impractical in clinical settings due to the lack of labeled data and the high computational cost. To address this, the authors propose CoTBox-TTT, a "plug-and-play" framework that adapts vision-language models at the moment of inference without requiring retraining or new ground-truth labels.

Methodology: Visual Chain-of-Thought Boxes

The core innovation of CoTBox-TTT is the use of bounding boxes as a visual "chain-of-thought" (CoT). Rather than generating an answer directly, the model is forced to identify the visual evidence supporting its conclusion.

Overview of the CoTBox-TTT framework, illustrating the dual-pass grounding mechanism and the use of soft prompts for test-time adaptation.

As shown in Figure 1, the framework employs three primary strategies:

  1. Dual-Pass Grounding: The model predicts a bounding box ($b_1$) on the original image and then validates this evidence on a localized crop ($I_{crop}$). This two-pass check ensures the model is looking at the correct anatomical or pathological features.
  2. Cross-view Consistency: The system utilizes an Exponential Moving Average (EMA) teacher—a stabilized version of the model—to guide a "student" version. The objective is to ensure that the generated answer remains consistent whether the model processes the full image or the zoomed-in crop.
  3. Parameter-Efficient Adaptation: To maintain clinical feasibility, the core backbones (the LLM and Vision Encoder) remain frozen. Only a small set of continuous soft prompts are updated during a brief 20-epoch adaptation phase at test time.

Experimental Setup and Datasets

The researchers evaluated CoTBox-TTT across three diverse medical VQA benchmarks to test its robustness:

  • VQA-RAD: Clinician-authored radiology images covering various body parts.
  • SLAKE: A bilingual (English/Chinese) radiology dataset integrated with a medical knowledge base.
  • PathVQA: A large-scale pathology dataset containing nearly 32,000 questions derived from digital libraries and textbooks.

Results and Impact

The implementation of CoTBox-TTT yielded substantial improvements across multiple model architectures, including general-purpose models like LLaVA and specialized medical models like Hulu-Med.

  • Accuracy Gains: Integrating CoTBox-TTT with LLaVA increased closed-ended accuracy by 12.3% on PathVQA and 8.09% on VQA-RAD.
  • Recall Improvements: For open-ended, free-form answers, LLaVA’s recall on PathVQA saw a dramatic increase from 7.74 to 32.60.
  • Interpretability: By requiring the model to output a bounding box, CoTBox-TTT provides an "evidence trail." This allows clinicians to verify which specific region of an image the AI used to reach its conclusion, significantly reducing the risk of acting on hallucinations.

Conclusion and Future Directions

CoTBox-TTT demonstrates that forcing a model to commit to specific visual evidence acts as a powerful regularizer during inference. While highly effective for 2D images, the authors note that future work should extend this framework to 3D volumes (such as CT or MRI scans) and explore pixel-level masks to better capture diffuse or multi-focal medical patterns. By prioritizing consistency and evidence-first design, CoTBox-TTT offers a scalable path toward more reliable AI in clinical diagnostics.

Summary updated

BibTeX
@misc{qian2025cotbox,
  author = {Jiahe Qian and Yuhao Shen and Zhangtianyi Chen and Juexiao Zhou and Peisong Wang},
  title = {{CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Training}},
  year = {2025},
  eprint = {2511.12446},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.12446},
}
Figure