Abstract
Self-supervised pretraining of visual encoders offers a promising avenue to reduce the dependency of Visual Question Answering (VQA) models on expensive human-annotated datasets. This paper evaluates two primary self-supervised paradigms: Contrastive Learning (CL) and Energy-Based Models (EBMs). Through experiments on a diagnostic synthetic dataset, we investigate their impact on systematic generalization—the ability to identify unseen combinations of attributes—and out-of-distribution (OOD) detection. Our findings reveal that while both methods outperform non-pretrained baselines, Contrastive Learning (specifically SimCLR) is significantly more stable and effective for generalization, achieving near-perfect accuracy. EBMs, while superior in specific OOD detection scenarios, suffer from training instabilities that limit their broader utility in VQA pipelines.
Introduction
Training robust VQA models typically necessitates massive datasets of image-question-answer triplets. This reliance not only incurs high costs but also embeds human biases into the models. While foundation models leverage self-supervision, they often require aligned image-caption pairs. This research asks whether pretraining a visual encoder on raw, unlabeled images alone can improve VQA performance, specifically focusing on systematic generalization to unseen attribute combinations (e.g., identifying a "red cube" after only seeing "blue cubes" and "red spheres" during training).
Methodology
The study compares two distinct self-supervised approaches:
- Contrastive Learning (SimCLR): This discriminative approach trains the model to map different augmented views of the same image to similar points in a latent space, while pushing views of different images apart. This encourages the learning of invariant and disentangled features.
- Energy-Based Models (EBMs): This generative approach assigns a scalar "energy" value to inputs, aiming to minimize energy for real data and maximize it for "fantasy" images generated via Markov Chain Monte Carlo (MCMC) sampling.
To isolate the effects of pretraining, the researchers employed a standard CNN-LSTM architecture, ensuring that performance gains were attributable to the pretraining paradigm rather than complex transformer dynamics.
Experimental Setup
The models were evaluated using a diagnostic toy dataset inspired by CLEVR, featuring geometric objects with varying shapes and colors. To test systematic generalization, a "CoGenT" style split was used: pretraining occurred on one set of colors, VQA training on a second, and testing on a third, entirely unseen combination. OOD detection was further tested using datasets like CIFAR-10, MNIST, and random noise.
Key Results and Discussion
The experimental results highlight a clear performance gap between the two methods in terms of both accuracy and reliability.
Generalization and Stability
Contrastive Learning emerged as the superior method for systematic generalization. By learning representations from unlabeled images, SimCLR effectively disentangled features like "shape" and "color," allowing the model to succeed on unseen combinations with ~99.3% accuracy. In contrast, EBMs were found to be highly unstable during training, with performance varying significantly across different training set sizes.
Out-of-Distribution Detection
While EBMs struggled with stability, they excelled at OOD detection for visually distinct data (e.g., noise or digits vs. 3D shapes), achieving near-perfect AUROC scores. However, their reliability diminished when OOD samples were visually similar to the training distribution, such as empty backgrounds from the same simulator.
Conclusion
This comparative study demonstrates that Contrastive Learning is currently the more effective and reliable choice for self-supervised pretraining in VQA, particularly for enhancing systematic generalization in low-data regimes. While EBMs offer unique advantages for OOD detection, their inherent training instabilities present a significant hurdle. Future research may explore hybrid models that combine the discriminative strength of CL with the generative awareness of EBMs.
Summary updated