QCAP-VQA: Question-Conditioned Capped Prior-Mass Supervision for Medical Visual Question Answering

Yining Zhao, Catarina Moreira, Mukesh Prasad, Ali BrayteearXiv 2026 · arXiv

In the high-stakes world of medical imaging, we don't just want a model to give the right answer; we want it to give the right answer for the right reasons. If a model correctly identifies a lung nodule but was actually looking at the patient's shoulder, that’s a "clever Hans" shortcut that could fail in a clinical setting.

The paper "QCAP-VQA: Question-Conditioned Capped Prior-Mass Supervision for Medical Visual Question Answering" tackles this problem of anatomical grounding in Medical Visual Question Answering (Med-VQA).

TL;DR

QCAP-VQA uses a novel "capped" loss function to supervise model attention using noisy, automatically generated anatomical priors. By only requiring a minimum threshold of attention on the relevant anatomy, it improves anatomical grounding without sacrificing answer accuracy or over-fitting to imperfect labels.

What Problem Does This Paper Address?

Med-VQA models often suffer from "shortcut learning," where they exploit statistical regularities in text or unrelated image features rather than truly "understanding" the medical image. While we can use spatial supervision to force models to look at specific regions, expert-annotated masks are expensive and rare.

Recent methods use "weak labels"—priors generated automatically by zero-shot segmentation models like MedSAM. However, these priors are noisy: they might be slightly off-center, too large, or miss parts of the anatomy. If you force a model to match a noisy prior perfectly (using standard losses like Mean Squared Error), you might actually be teaching it the wrong thing, leading to a drop in overall accuracy.

Methods

The authors propose a three-stage framework called QCAP-VQA:

  1. Stage A (Base Training): A standard Med-VQA model is trained using a vision-language backbone (BiomedCLIP and PubMedBERT).
  2. Stage B (Head Allocation): Not all attention heads are created equal. The authors use a "knockout" analysis—temporarily disabling heads to see which ones hurt accuracy the most—to select a subset of $K$ cross-attention heads for supervision.
  3. Stage C (Capped Supervision): This is the core innovation. Instead of a loss that keeps pushing for more attention on the prior, they use a Capped Prior-Mass objective.

As shown in the overview below, the model only receives a penalty if the attention mass inside the prior falls below a threshold $\tau$ (e.g., 40%). Once that threshold is met, the spatial loss becomes zero, allowing the model's primary VQA objective to fine-tune the attention distribution without further "spatial pressure."

Fig. 1. Overview of QCAP-VQA. Stage A trains the VQA model, Stage B selects K heads within each run, and Stage C applies capped prior-mass supervision to them (Lcap: Latt of Eq. (4)). Hvis/Hsem: constrained and unconstrained heads, not fixed head functions.

Datasets

  • SLAKE-EN: Used for training and the primary evaluation. Crucially, the authors use SLAKE’s expert-annotated organ and lesion boxes for independent evaluation only—these boxes are never seen during training.
  • VQA-RAD: Used as a secondary benchmark to compare against state-of-the-art methods.

Key Insights and Ideas

  • The "Capped" Advantage: Traditional "pooled" objectives keep rewarding the model for concentrating attention inside the prior until it reaches nearly 100%. If the prior is noisy, this "over-concentration" hurts accuracy. The capped objective acts like a "good enough" constraint, providing stability across a wide range of regularization strengths.
  • The Evaluation Trap: The authors found a startling Spearman correlation of -0.87 between rankings based on the training prior versus rankings based on expert ground truth. In other words, models that seem to "obey" the noisy training prior the best are often the ones that align worst with actual human experts. This highlights the danger of evaluating models against the same noisy labels used for training.
  • Conditioning Trade-off: There is a subtle trade-off between "looking at the right place" and "shifting focus correctly." As you increase the number of constrained heads, the model puts more total mass on the named anatomy, but it becomes less effective at shifting its attention when the question changes (e.g., moving from a question about the liver to one about the spleen on the same image).

Key Results

  • Accuracy: QCAP-VQA matches the accuracy of a baseline without spatial supervision (84.26% vs 84.23% on SLAKE) while significantly increasing anatomical grounding.
  • Stability: Over a 64-fold range of regularization strength ($\lambda$), the capped objective kept the attention mass on the prior stable (between 0.67 and 0.77). In contrast, the "pooled" objective drove mass to 0.97 but suffered a 1.57 percentage point drop in accuracy.
  • Qualitative Grounding: As seen in Figure 5, the capped model (right column) shows much cleaner, more focused attention on the relevant organs compared to models without supervision or those using standard MSE.
Qualitative comparison of attention maps for liver and spleen questions.

Limitations and Future Work

  • Small Evaluation Cohort: The detailed question-conditioning analysis was limited to 15 question pairs from 13 images due to the requirement for unambiguous question-region correspondence in the SLAKE dataset.
  • Prior Quality: The method still relies on the existence of a somewhat reasonable automatic prior. If the segmentation model fails completely, the supervision will be misleading.
  • Future Directions: The authors suggest that larger annotated cohorts and even more accurate zero-shot priors are needed to fully assess how these models generalize across diverse clinical scenarios. They also point toward investigating why "more heads" doesn't always lead to better question-conditioned shifts.

Summary updated

Abstract

Medical visual question answering models can answer correctly without attending to the anatomy that the question names. Automatic region priors allow spatial supervision without expert masks, but they are noisy, and it is unclear how strongly they should shape attention and how their effect should be verified. We present QCAP-VQA, which supervises a subset of cross-attention heads with a question-conditioned prior built from ranked segmentation proposals. Its capped prior-mass objective penalizes insufficient attention inside the prior only below a threshold. We evaluate attention against SLAKE structure boxes that play no role in training, separating attention on the named anatomy from how attention shifts between questions about the same image. QCAP-VQA matches the accuracy of a baseline without spatial supervision (84.26% vs. 84.23%), and every evaluated prior-supervised model places more attention on the annotated structures. Over a 64-fold range of regularization strength, the capped objective keeps prior mass between 0.67 and 0.77, whereas a pooled region-mass objective drives it to 0.97 and loses accuracy. The cost lies in question conditioning: as the constraint widens or strengthens, attention shifts agree less with the expert-defined contrast. Across 13 configurations, prior-referenced and expert-referenced evaluation rank the models in largely opposite order (Spearman correlation of -0.87), and the capped models it favors track the prior's own errors. Because clinical use of medical visual question answering requires answers grounded in the queried anatomy, supervision from automatic priors should be judged against independent anatomical references.
BibTeX
@misc{zhao2026qcap,
  author = {Yining Zhao and Catarina Moreira and Mukesh Prasad and Ali Braytee},
  title = {{QCAP-VQA: Question-Conditioned Capped Prior-Mass Supervision for Medical Visual Question Answering}},
  year = {2026},
  eprint = {2611.17708},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17708},
}
Figure