EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering

Violetta Shevchenko, Ehsan Abbasnejad, Anthony R. Dick, Anton van den Hengel, Damien TeneyarXiv 2022 · DOI · arXiv

Abstract

Self-supervised pretraining of visual encoders offers a promising avenue to reduce the dependency of Visual Question Answering (VQA) models on expensive human-annotated datasets. This paper evaluates two primary self-supervised paradigms: Contrastive Learning (CL) and Energy-Based Models (EBMs). Through experiments on a diagnostic synthetic dataset, we investigate their impact on systematic generalization—the ability to identify unseen combinations of attributes—and out-of-distribution (OOD) detection. Our findings reveal that while both methods outperform non-pretrained baselines, Contrastive Learning (specifically SimCLR) is significantly more stable and effective for generalization, achieving near-perfect accuracy. EBMs, while superior in specific OOD detection scenarios, suffer from training instabilities that limit their broader utility in VQA pipelines.

Introduction

Training robust VQA models typically necessitates massive datasets of image-question-answer triplets. This reliance not only incurs high costs but also embeds human biases into the models. While foundation models leverage self-supervision, they often require aligned image-caption pairs. This research asks whether pretraining a visual encoder on raw, unlabeled images alone can improve VQA performance, specifically focusing on systematic generalization to unseen attribute combinations (e.g., identifying a "red cube" after only seeing "blue cubes" and "red spheres" during training).

Methodology

The study compares two distinct self-supervised approaches:

  1. Contrastive Learning (SimCLR): This discriminative approach trains the model to map different augmented views of the same image to similar points in a latent space, while pushing views of different images apart. This encourages the learning of invariant and disentangled features.
  2. Energy-Based Models (EBMs): This generative approach assigns a scalar "energy" value to inputs, aiming to minimize energy for real data and maximize it for "fantasy" images generated via Markov Chain Monte Carlo (MCMC) sampling.
The Contrastive Learning (SimCLR) pipeline uses data augmentation and a projection head to learn discriminative features.

To isolate the effects of pretraining, the researchers employed a standard CNN-LSTM architecture, ensuring that performance gains were attributable to the pretraining paradigm rather than complex transformer dynamics.

Experimental Setup

The models were evaluated using a diagnostic toy dataset inspired by CLEVR, featuring geometric objects with varying shapes and colors. To test systematic generalization, a "CoGenT" style split was used: pretraining occurred on one set of colors, VQA training on a second, and testing on a third, entirely unseen combination. OOD detection was further tested using datasets like CIFAR-10, MNIST, and random noise.

Key Results and Discussion

The experimental results highlight a clear performance gap between the two methods in terms of both accuracy and reliability.

Generalization and Stability

Contrastive Learning emerged as the superior method for systematic generalization. By learning representations from unlabeled images, SimCLR effectively disentangled features like "shape" and "color," allowing the model to succeed on unseen combinations with ~99.3% accuracy. In contrast, EBMs were found to be highly unstable during training, with performance varying significantly across different training set sizes.

Comparison of test accuracy between EBM and SimCLR across different training set sizes, highlighting the stability and performance of Contrastive Learning.

Out-of-Distribution Detection

While EBMs struggled with stability, they excelled at OOD detection for visually distinct data (e.g., noise or digits vs. 3D shapes), achieving near-perfect AUROC scores. However, their reliability diminished when OOD samples were visually similar to the training distribution, such as empty backgrounds from the same simulator.

Conclusion

This comparative study demonstrates that Contrastive Learning is currently the more effective and reliable choice for self-supervised pretraining in VQA, particularly for enhancing systematic generalization in low-data regimes. While EBMs offer unique advantages for OOD detection, their inherent training instabilities present a significant hurdle. Future research may explore hybrid models that combine the discriminative strength of CL with the generative awareness of EBMs.

Summary updated

Abstract

The availability of clean and diverse labeled data is a major roadblock for training models on complex tasks such as visual question answering (VQA). The extensive work on large vision-and-language models has shown that self-supervised learning is effective for pretraining multimodal interactions. In this technical report, we focus on visual representations. We review and evaluate self-supervised methods to leverage unlabeled images and pretrain a model, which we then fine-tune on a custom VQA task that allows controlled evaluation and diagnosis. We compare energy-based models (EBMs) with contrastive learning (CL). While EBMs are growing in popularity, they lack an evaluation on downstream tasks. We find that both EBMs and CL can learn representations from unlabeled images that enable training a VQA model on very little annotated data. In a simple setting similar to CLEVR, we find that CL representations also improve systematic generalization, and even match the performance of representations from a larger, supervised, ImageNet-pretrained model. However, we find EBMs to be difficult to train because of instabilities and high variability in their results. Although EBMs prove useful for OOD detection, other results on supervised energy-based training and uncertainty calibration are largely negative. Overall, CL currently seems a preferable option over EBMs.
BibTeX
@misc{shevchenko2022ebms,
  author = {Violetta Shevchenko and Ehsan Abbasnejad and Anthony R. Dick and Anton van den Hengel and Damien Teney},
  title = {{EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering}},
  year = {2022},
  eprint = {2206.14355},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.14355},
}
Figure