Counterfactual Vision-Language Data Synthesis with Intra-Sample Contrast Learning

Zhecan Wang, Yicheng He, Wenhao Li, Haoxuan You, Long Chen, Noel C Codella, Yulei Niu, Kai-Wei Chang, Shih-Fu ChangICLR 2023 · OpenReview

Abstract

Existing Visual Learning (VL) benchmarks often contain exploitative biases. Most former works only attempted to mitigate biases in semantically low-level and conventional visual-question-answering typed datasets like VQA and GQA. However, these methods cannot generalize to recently emerging highly semantic VL datasets like VCR and are also difficult to scale due to many severe problems like high-cost labors, drastically disrupting the data distribution\textit{, etc.}To resolve those problems and also address other unique biases on VCR-like datasets, we first conduct in-depth analysis and identify important biases in VCR dataset. We further propose a generalized solution that synthesizes counterfactual image and text data based on the original query's semantic focus while producing less distortion to the data distribution. To utilize our synthesized data, we also design an innovative intra-sample contrastive training strategy to assist QA learning in Visual Commonsense Reasoning (VCR). Moreover, our synthesized VL data also serve as a highly-semantic debiased benchmark for evaluating future VL models' robustness. Extensive experiments show that our proposed synthesized data and training strategy improve existing VL models' performances on both the original VCR dataset and our proposed debiased benchmark.
BibTeX
@inproceedings{wang2023counterfactual,
  author = {Zhecan Wang and Yicheng He and Wenhao Li and Haoxuan You and Long Chen and Noel C Codella and Yulei Niu and Kai-Wei Chang and Shih-Fu Chang},
  title = {{Counterfactual Vision-Language Data Synthesis with Intra-Sample Contrast Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=K1NKDaNM9i},
}

Loading related content...

Figure