Robust Visual Question Answering With Contrastive-Adversarial Consistency Constraints

Jiawei Zhu, Meirong Ding, Yishu Liu, Biqing Zeng, Guangming Lu, Bingzhi ChenICME 2024 · OpenReview

Abstract

Visual cues and question semantics contribute to final answer predictions from distinct perspectives. However, inherent language bias confounds the relationship between visual and question cues, leading to a misguided preference for question semantics. Different from the existing studies that focus on inter-class discrimination, this paper proposes a robust visual question answering framework with contrastive-adversarial consistency constraints (CACC) at both inter- and intra-instance levels. From a fine-grained instance-level perspective, our approach initially introduces an effective inter-instance contrastive constraint to perform adaptive bias rectification. To enhance intra-instance invariance and reduce information redundancy, we refine the concept of semantic structure relationships by constructing intra-instance adversarial constraints using the Hilbert-Schmidt Independence Criterion (HSIC) independence criterion. Benefitting from both inter- and intra-instance perspectives, our method can effectively alleviate these language biases, enhancing the overall robustness of the representation. Extensive experiments on multiple benchmark datasets consistently demonstrate the superiority of our CACC over state-of-the-art baselines.
BibTeX
@inproceedings{zhu2024robust,
  author = {Jiawei Zhu and Meirong Ding and Yishu Liu and Biqing Zeng and Guangming Lu and Bingzhi Chen},
  title = {{Robust Visual Question Answering With Contrastive-Adversarial Consistency Constraints}},
  booktitle = {ICME},
  year = {2024},
  url = {https://openreview.net/forum?id=rkVMxNPXA4},
}

Loading related content...

Figure