Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning

Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, Alan Yuille

TL;DR

Super-CLEVR is a new virtual benchmark designed to isolate and diagnose how four distinct factors of domain shift—visual complexity, question redundancy, concept distribution, and concept compositionality—affect Visual Question Answering (VQA) models. To combat these shifts, the authors introduce Probabilistic NSVQA (P-NSVQA), a neuro-symbolic model that accounts for visual uncertainty to deliver superior robustness across out-of-distribution scenarios.

What Problem Does This Paper Address?

Visual Question Answering (VQA) models are notorious for failing when tested on data outside their training distribution. Because VQA requires a mix of computer vision and natural language processing, multiple factors of variation are constantly intertwined. When a model fails on a new dataset, it is nearly impossible to pinpoint why: Is it because the images are visually busier? Are the questions phrased with more redundancy? Or has the underlying distribution of concepts shifted?

Current benchmarks cannot answer these questions because they change all variables at once. This paper introduces Super-CLEVR, a controllable virtual dataset that decomposes domain shifts into isolated axes, allowing researchers to independently analyze and diagnose model robustness.

Figure 1. We decompose VQA domain shifts into four contributing factors: visual complexity, question redundancy, concept distribution and concept compositionality. The domain shifts along each factor can be independently studied with the proposed Super-CLEVR dataset.

Methods

The paper introduces both a benchmarking framework and a novel algorithmic improvement:

  1. Super-CLEVR Control Pipeline: The authors generate synthetic data splits where only one factor of domain shift changes at a time. For example, they can keep the exact same questions and scene layouts but increase visual complexity by adding textures and object parts.
  2. Probabilistic NSVQA (P-NSVQA): Traditional neuro-symbolic VQA models parse an image into a deterministic scene graph and execute a symbolic program step-by-step. If the initial scene parser makes a mistake, the error propagates catastrophically. P-NSVQA replaces this rigid execution with a probabilistic reasoning process. By treating scene-parsing confidence scores as likelihood functions, it calculates the joint probability of the entire reasoning chain, allowing the model to recover from visual ambiguities.

Datasets

  • Super-CLEVR (New): Contains 30k images per variant (20k train, 5k val, 5k test). Unlike the original CLEVR dataset's simple geometric shapes, Super-CLEVR uses 21 complex 3D vehicle models across 5 categories (car, bus, airplane, bike, motorcycle) from the UDA-Part dataset, featuring detailed part annotations (e.g., wheels, doors).
  • CLEVR: Used as the baseline for standard, low-complexity in-domain evaluation.
  • GQA: A real-world VQA dataset used to validate that the question-redundancy insights discovered in Super-CLEVR successfully transfer to real-world data.
Figure 2. Super-CLEVR contains 21 vehicle models belonging to 5 categories, with controllable attributes.

Key Insights and Ideas

  • The Modular Trade-Off: Neural-modular and symbolic models (like NSCL and NSVQA) are highly robust against question redundancy shifts. Because they reason step-by-step, they don't easily exploit spurious linguistic shortcuts. However, they struggle with visual complexity because simple symbolic modules fail to capture intricate visual details.
  • Non-Modular Strengths and Weaknesses: Dense, non-modular models (like FiLM and mDETR) handle visual complexity well due to their distributed representations, but their performance plummets when questions contain redundant descriptors or when concept distributions shift.
  • Uncertainty Bridges the Gap: By injecting probabilistic uncertainty into a symbolic executor, P-NSVQA mitigates the primary weakness of neuro-symbolic models (visual vulnerability) while preserving their superior reasoning capabilities.

Key Results

  • Benchmark Difficulty: Super-CLEVR proves to be significantly more challenging than the original CLEVR dataset. Even the state-of-the-art transformer model, mDETR, drops from 97.52% accuracy on CLEVR to 82.65% on Super-CLEVR.
  • P-NSVQA Performance: P-NSVQA achieves the highest in-domain accuracy on Super-CLEVR at 95.76%, outperforming its deterministic baseline by nearly 3%.
  • Out-of-Domain Robustness: To evaluate robustness, the authors measured Relative Degrade (RD)—the percentage of accuracy lost under a domain shift. P-NSVQA achieved the lowest degradation on three out of four factors, showing remarkable resilience to question redundancy (0.84% RD) and concept distribution shifts (13.72% RD).
Figure 3. Comparison of models’ accuracy on Super-CLEVR and the original CLEVR dataset.
Table 1. Accuracy of models trained and tested on different domains. Column headings indicate training settings, while rows indicate the dataset variant for testing. The best performance in each row (i.e. the best training setting) is marked in bold and best performance in each column (i.e. the best testing setting) is underlined. Description for different splits is in Sec. 3.3 and analysis is in Sec. 5.2.

Limitations and Future Work

The primary limitation is the synthetic nature of the Super-CLEVR dataset. While rendering 3D models provides perfect ground-truth control over domain factors, synthetic data cannot fully replicate the nuances of real-world imagery.

The authors note that future work should focus on translating these controlled diagnostic techniques into real-world datasets. Additionally, they suggest Super-CLEVR can be expanded to study hierarchical, part-based visual reasoning—an area where current VQA models still severely struggle.

Summary updated

Abstract

Visual Question Answering (VQA) models often perform poorly on out-of-distribution data and struggle on domain generalization. Due to the multi-modal nature of this task, multiple factors of variation are intertwined, making generalization difficult to analyze. This motivates us to introduce a virtual benchmark, Super-CLEVR, where different factors in VQA domain shifts can be isolated in order that their effects can be studied independently. Four factors are considered: visual complexity, question redundancy, concept distribution and concept compositionality. With controllably generated data, Super-CLEVR enables us to test VQA methods in situations where the test data differs from the training data along each of these axes. We study four existing methods, including two neural symbolic methods NSCL and NSVQA, and two non-symbolic methods FiLM and mDETR; and our proposed method, probabilistic NSVQA (P-NSVQA), which extends NSVQA with uncertainty reasoning. P-NSVQA outperforms other methods on three of the four domain shift factors. Our results suggest that disentangling reasoning and perception, combined with probabilistic uncertainty, form a strong VQA model that is more robust to domain shifts. The dataset and code are released at https://github.com/Lizw14/Super-CLEVR.
BibTeX
@inproceedings{li2022super,
  author = {Zhuowan Li and Xingrui Wang and Elias Stengel-Eskin and Adam Kortylewski and Wufei Ma and Benjamin Van Durme and Alan Yuille},
  title = {{Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure