When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach

Feifei Zhang, Zhaoyi Zhang, Xi Zhang, Changsheng XuAAAI 2025 · DOI

Visual Question Answering (VQA) has traditionally been treated as a classification task within a fixed vocabulary. While models excel at selecting answers from predefined lists, they often fail in "open-world" scenarios involving rare objects or novel concepts. This paper, presented at AAAI-25, introduces the Open-Vocabulary VQA (OVVQA) benchmark and a Causal Adapter to move beyond statistical memorization toward true visual reasoning.

The Challenge: Canonical Class Overfitting

Most VQA benchmarks operate in a closed-set paradigm, where models learn to exploit statistical correlations in the training data. This leads to canonical class overfitting bias. For instance, if a dataset frequently pairs "person" with "motorcycle," a model might correctly identify a person on a bike during training. However, when presented with a "skeleton man" on a motorcycle—a novel class—the biased model defaults to "person" because it relies on common correlations rather than specific visual evidence.

Comparison of closed-set vs. open-vocabulary VQA and the impact of dataset bias.

Standard fine-tuning methods often exacerbate this issue, improving performance on common ("base") answers while causing a catastrophic drop in performance on rare ("novel") ones.

Proposed Method: The Causal Adapter

To address these biases, the researchers propose a two-pronged technical solution:

1. Causal Adapter with Front-Door Adjustment

Instead of standard fine-tuning, a plug-and-play module is inserted between the encoder and decoder. This adapter implements front-door adjustment, a concept from causal inference. By introducing a "mediator" variable, the model breaks the spurious links between images and answers caused by hidden dataset confounders. Through stratified visual features and cross-sample sampling, the model learns the genuine causal effect of visual input on the generated answer.

2. Adaptive Transfer Loss ($L_{ATL}$)

To prevent the loss of pre-trained knowledge during fine-tuning, the authors introduce a nonlinear, Transformer-based approach to combine auxiliary losses, such as image-text matching and masked language modeling. This ensures the model retains its foundational capabilities while adapting to the VQA task.

Architecture of the proposed model featuring the Causal Adapter and causal graph representations.

The OVVQA Benchmark

The authors reconstructed three standard datasets to create the OVVQA benchmark: OV-VQAv2, OV-GQA, and OV-OKVQA. In these versions, answers are split into "base" (high-frequency) and "novel" (unseen) classes. Models are trained only on base classes but tested on both, strictly evaluating their ability to generalize to novel categories.

Experimental Results

The Causal Adapter was evaluated using established backbones like VL-T5 and VL-BART. Key findings include:

  • Novel Class Improvement: On OV-VQAv2, novel class accuracy rose from a 28.29% baseline to 32.69%.
  • Balanced Generalization: The Harmonic Mean (H)—which balances base and novel performance—improved by nearly 6 points (from 38.53 to 44.44).
  • Robustness: The approach consistently outperformed prominent models like LXMERT and ALBEF across all three benchmarks, demonstrating that causal intervention effectively mitigates dataset bias without sacrificing accuracy on common classes.

Conclusion

The Causal Adapter represents a significant step toward robust, open-vocabulary VQA. By prioritizing causality over simple correlation, the model avoids the "Skeleton Man" pitfall, looking at specific visual features rather than relying on biased priors. While currently focused on visual-answer relationships, future work aims to extend this causal framework to more complex multimodal reasoning tasks.

Reference: Zhang, F., et al. (2025). When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach. Proceedings of the 39th AAAI Conference on Artificial Intelligence.

Summary updated

BibTeX
@inproceedings{zhang2025when,
  author = {Feifei Zhang and Zhaoyi Zhang and Xi Zhang and Changsheng Xu},
  title = {{When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.33072},
}
Figure