ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models

Duy M. H. Nguyen, Nghiem T. Diep, Trung Q. Nguyen, Hoang-Bao Le, Tai Nguyen, Tien Nguyen, Nhat Ho, Pengtao Xie, Roger Wattenhofer, James Zou, Daniel Sonntag, Mathias Niepert

TL;DR

Medical vision-language models typically require massive amounts of expensive instruction-following data to align clinical images with text. ExGra-Med overcomes this bottleneck by introducing a multi-graph alignment framework that pairs images, instructions, and LLM-enriched extended captions using a scalable barycenter graph. The result is a highly efficient model that matches state-of-the-art performance on medical visual question-answering (VQA) tasks using just 10% of the standard pre-training data.

What Problem Does This Paper Address?

State-of-the-art medical multi-modal Large Language Models (med-MLLMs), such as LLaVA-Med, rely heavily on autoregressive training objectives (predicting the next token) to learn how to follow instructions based on medical images. However, the authors reveal a critical flaw in this paradigm: autoregressive learning is incredibly data-hungry.

Without massive volumes of medical instruction-following (IF) data, the alignment between the vision encoder and the language decoder degrades rapidly. For instance, when LLaVA-Med is trained on only 10% of its pre-training data, its downstream performance on medical VQA tasks plummets drastically.

Figure 1: Our EXGRA-MED versus LLaVA-Med across varying instruction-following (IF) pre-training data sizes, highlighting the data-hungry behavior of auto-regressive modeling. Both models are fine- tuned on the same VQA-RAD training set after the pre-training stage at each IF rate. At 100% IF pre- training, ExGra-Med and LLaVA-Med are bench- marked against other state-of-the-art models, all fine- tuned on the same VQA-RAD training set (except GPT- 4, which is evaluated without fine-tuning). Circle ra- dius represents the number of model parameters.

Because curating high-quality, expert-annotated medical IF data is exceptionally costly and time-consuming, there is an urgent need for sample-efficient training methods that don't sacrifice clinical accuracy.

Methods

ExGra-Med addresses this data scarcity by introducing a structure-aware multi-graph alignment framework that optimizes cross-modal coherence in the latent space.

The framework consists of three core components:

  1. Extended Context Enrichment: To provide richer supervision, the authors use a frozen LLM (GPT-4o) to generate contextually extended, semantically rich versions of the original answers without altering their clinical intent.
Figure 2: Illustration for creating the ex- tended context instruction-following data powered by GPT-4o.
  1. Barycenter Graph Alignment: Instead of performing computationally expensive pairwise alignments across multiple modalities (which scales poorly), ExGra-Med constructs three modality-specific graphs: one for visual features ($G_v$), one for original answers ($G_a$), and one for extended answers ($G_{ae}$). These graphs are then aligned to a central, shared barycenter graph ($G_{br}$). This reformulates a complex combinatorial problem into separate, scalable graph-matching tasks.
  2. Black-Box Gradient Estimation: Combinatorial graph matching is typically non-differentiable, making backpropagation through a 7-billion-parameter LLM impossible with standard gradient descent. ExGra-Med bypasses this by using Implicit Maximum Likelihood Estimation (IMLE), a technique that estimates gradients using noise-perturbed solutions, enabling efficient end-to-end training.
Figure 3: Overview of EXGRA-MED: The large language model gσ and the projector hϕ are trained jointly by aligning a triplet of modalities - input image, instruction-following data, and extended captions - through a structure-aware multigraph alignment (Eq.(3)). This alignment operates over graphs Gv, Ga, and Gae, representing the visual, instruction, and extended textual information, respectively, via a shared barycenter graph. The entire model is optimized end-to-end using modern black-box gradient estimation techniques to enable efficient learning across modalities [70, 63].

Datasets

The model was pre-trained and evaluated on several prominent biomedical datasets:

  • Pre-training: A subset of the PMC-15M dataset, utilizing 600K image-text pairs for initial alignment and 60K multi-round instruction-following examples generated via GPT-4.
  • Downstream Evaluation (Medical VQA): Evaluated on VQA-RAD (radiology), SLAKE (multimodal/multilingual radiology), and PathVQA (pathology).
  • Zero-Shot Classification: Benchmarked using the OmniMedVQA benchmark, spanning 23 distinct datasets across Microscopy, Computed Tomography (CT), and Chest X-Ray (CXR) modalities.

Key Insights and Ideas

  • Global vs. Local Alignment: Traditional contrastive learning aligns individual image-caption pairs locally. ExGra-Med enforces global structural graph constraints across the entire dataset batch. This preserves relational consistency, which is vital in medical imaging where different diseases can present with highly similar visual features.
  • The Power of Paraphrastic Context: Simply augmenting data isn't enough. By forcing the model to align visual features with both concise answers and contextually enriched descriptions, the model develops a deeper, more robust semantic grounding that transfers better to downstream tasks.
  • Algorithmic Efficiency Over Scale: The paper challenges the "bigger is always better" trend in AI. It demonstrates that refining how a model aligns modalities is just as crucial as scaling up model parameters or data volume.

Key Results

  • Data Efficiency: When restricted to just 10% of pre-training data, ExGra-Med achieved a 72.52% score on VQA-RAD, effectively matching the performance of LLaVA-Med trained on 100% of the data (72.64%), and outperforming the 10% LLaVA-Med baseline by 20.13%.
  • State-of-the-Art Performance: When scaled to full data, ExGra-Med consistently outperformed much larger models. Despite having only 7B parameters, it surpassed the 40B-parameter Med-Dr model across all evaluated VQA datasets.
  • Zero-Shot Generalization: On the OmniMedVQA benchmark, ExGra-Med outperformed specialized baselines like RadFM and Med-Flamingo, notably exceeding RadFM's accuracy in microscopy tasks by 8.2%.

Limitations and Future Work

While highly effective, the authors note a few limitations and areas for future exploration:

  • Architecture Constraints: The current implementation was primarily validated using the LLaVA architecture. Future work needs to verify its adaptability to other multimodal frameworks like Flamingo.
  • Domain-Specific Foundations: The framework currently utilizes general-purpose base models (LLaMA-7B). Incorporating LLMs and vision encoders specifically pre-trained on medical corpora could yield even higher clinical accuracy.
  • Reasoning Chains: The authors suggest extending this multi-graph alignment mechanism to medical visual Chain-of-Thought (CoT) reasoning to improve the explainability and trustworthiness of AI diagnostic assistants.

Summary updated

Abstract

State-of-the-art medical multi-modal LLMs (med-MLLMs), such as LLaVA-Med and BioMedGPT, primarily depend on scaling model size and data volume, with training driven largely by autoregressive objectives. However, we reveal that this approach can lead to weak vision-language alignment, making these models overly dependent on costly instruction-following data. To address this, we introduce ExGra-Med, a novel multi-graph alignment framework that jointly aligns images, instruction responses, and extended captions in the latent space, advancing semantic grounding and cross-modal coherence. To scale to large LLMs (e.g., LLaMA-7B), we develop an efficient end-to-end training scheme using black-box gradient estimation, enabling fast and scalable optimization. Empirically, ExGra-Med matches LLaVA-Med's performance using just 10% of the pre-training data, achieving a 20.13% gain on VQA-RAD and approaching full-data performance. It also outperforms strong baselines like BioMedGPT and RadFM on visual chatbot and zero-shot classification tasks, demonstrating its promise for efficient, high-quality vision-language integration in medical AI.
BibTeX
@inproceedings{nguyen2025exgra,
  author = {Duy M. H. Nguyen and Nghiem T. Diep and Trung Q. Nguyen and Hoang-Bao Le and Tai Nguyen and Tien Nguyen and TrungTin Nguyen and Nhat Ho and Pengtao Xie and Roger Wattenhofer and James Zou and Daniel Sonntag and Mathias Niepert},
  title = {{ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=TjWdyVWBAG},
}
Figure