Tri-VQA: Triangular Reasoning Medical Visual Question Answering for Multi-Attribute Analysis

Lin Fan, Xun Gong, Cenyang Zheng, Yafei Ou

Introduction

Medical Visual Question Answering (Med-VQA) is a critical intersection of computer vision and natural language processing, designed to assist clinicians by interpreting complex medical imagery—such as ultrasounds, MRIs, and CT scans—to answer diagnostic questions. Despite the rapid advancement of general-purpose VQA, medical applications face a significant "credibility gap." Standard models often arrive at correct answers through coincidental correlations rather than sound medical logic. Tri-VQA addresses this by introducing a triangular reasoning framework that incorporates reverse causal inference, ensuring models are right for the right reasons.

The Credibility Gap in Medical VQA

Most existing Med-VQA models rely on a "joint embedding" approach. In this paradigm, image features ($V$) and question features ($Q$) are fused into a single representation to predict an answer ($A$), following the linear logic of $V + Q \rightarrow A$.

Comparison of the standard Joint Embedding approach versus the proposed Triangular Reasoning framework.

While computationally efficient, this one-way street allows models to exploit dataset biases. For instance, if a dataset contains many "malignant" labels for a specific ultrasound texture, the model might learn that texture-label association without actually "understanding" the clinical attributes (like boundary or shape) that define malignancy. This lack of transparency is a major hurdle for clinical adoption. Furthermore, few models are specifically optimized for fine-grained tumor attributes, which are essential for nuanced diagnosis.

The Tri-VQA Framework

To bridge this gap, Tri-VQA moves beyond simple joint embedding to a multi-directional "triangular" reasoning structure. Motivated by causal effects, the framework treats the relationship between the image, question, and answer as a mutually inferable loop.

The detailed Tri-VQA architecture, illustrating the integration of forward and reverse causal inference paths.

The architecture consists of three primary paths:

  1. Forward Inference ($V + Q \rightarrow A$): This is the standard predictive path. The model utilizes a pre-trained CLIP model for visual feature extraction and BERT for textual encoding to generate an initial answer.
  2. Reverse Inference Q ($A + V \rightarrow Q$): The model is challenged to reconstruct the original question given the predicted answer and the image.
  3. Reverse Inference V ($A + Q \rightarrow V$): The model attempts to speculate on the visual features of the image given the answer and the question.

To ensure these reverse paths contribute to the final decision, the researchers introduced Secondary Forward Reasoning (SFR). In this step, the features generated during reverse inference are fed back through the forward path. If the model can navigate this entire triangle—predicting an answer, reconstructing the inputs, and then re-verifying the answer—it demonstrates a cohesive and stable understanding of the medical context.

Experimental Evaluation

The researchers evaluated Tri-VQA across three diverse datasets:

  • EUS Multi-Attribute Dataset: A specialized collection of 7,021 Endoscopic Ultrasound images annotated with six critical tumor attributes (Echo, Boundary, Shape, etc.).
  • VQA-RAD: A public dataset of clinically generated questions regarding radiology images.
  • SLAKE: A semantically-labeled, knowledge-enhanced medical VQA dataset.

The study focused on open-ended questions, which require deeper semantic understanding than binary yes/no queries.

Key Results and Insights

Tri-VQA established new state-of-the-art (SOTA) benchmarks across all tested datasets. On the EUS Dataset, it achieved an accuracy of 0.957 and an AUC of 0.935, significantly outperforming existing multi-attribute models like MAA-Net. On VQA-RAD and SLAKE, it reached accuracies of 0.738 and 0.831, respectively.

Ablation studies confirmed that the "triangle" is the primary driver of performance; removing either the Reverse Inference (RI) or the Secondary Forward Reasoning (SFR) components led to a measurable decline in accuracy.

Beyond raw performance, the framework provides a built-in "Trust Meter." The researchers found that the accuracy of the reverse inference (measured by how closely the reconstructed features match the ground truth) serves as a reliable indicator of the answer's correctness. When the model can accurately "explain" the question and image that led to its answer, that answer is statistically much more likely to be correct.

Conclusion

Tri-VQA represents a shift from black-box prediction to logical consistency in medical AI. By enforcing causal stability through triangular reasoning, the framework not only improves diagnostic accuracy but also provides a pathway toward more interpretable and reliable AI assistants in clinical settings. Future work will focus on refining feature extraction for highly similar tumor characteristics and integrating these reliability indicators into real-time clinical interfaces.

Summary updated

BibTeX
@inproceedings{fan2024tri,
  author = {Lin Fan and Xun Gong and Cenyang Zheng and Yafei Ou},
  title = {{Tri-VQA: Triangular Reasoning Medical Visual Question Answering for Multi-Attribute Analysis}},
  booktitle = {BIBM},
  year = {2024},
  doi = {10.1109/bibm62325.2024.10822825},
}
Figure