TL;DR
Super-CLEVR is a new virtual benchmark designed to isolate and diagnose how four distinct factors of domain shift—visual complexity, question redundancy, concept distribution, and concept compositionality—affect Visual Question Answering (VQA) models. To combat these shifts, the authors introduce Probabilistic NSVQA (P-NSVQA), a neuro-symbolic model that accounts for visual uncertainty to deliver superior robustness across out-of-distribution scenarios.
What Problem Does This Paper Address?
Visual Question Answering (VQA) models are notorious for failing when tested on data outside their training distribution. Because VQA requires a mix of computer vision and natural language processing, multiple factors of variation are constantly intertwined. When a model fails on a new dataset, it is nearly impossible to pinpoint why: Is it because the images are visually busier? Are the questions phrased with more redundancy? Or has the underlying distribution of concepts shifted?
Current benchmarks cannot answer these questions because they change all variables at once. This paper introduces Super-CLEVR, a controllable virtual dataset that decomposes domain shifts into isolated axes, allowing researchers to independently analyze and diagnose model robustness.
Methods
The paper introduces both a benchmarking framework and a novel algorithmic improvement:
- Super-CLEVR Control Pipeline: The authors generate synthetic data splits where only one factor of domain shift changes at a time. For example, they can keep the exact same questions and scene layouts but increase visual complexity by adding textures and object parts.
- Probabilistic NSVQA (P-NSVQA): Traditional neuro-symbolic VQA models parse an image into a deterministic scene graph and execute a symbolic program step-by-step. If the initial scene parser makes a mistake, the error propagates catastrophically. P-NSVQA replaces this rigid execution with a probabilistic reasoning process. By treating scene-parsing confidence scores as likelihood functions, it calculates the joint probability of the entire reasoning chain, allowing the model to recover from visual ambiguities.
Datasets
- Super-CLEVR (New): Contains 30k images per variant (20k train, 5k val, 5k test). Unlike the original CLEVR dataset's simple geometric shapes, Super-CLEVR uses 21 complex 3D vehicle models across 5 categories (car, bus, airplane, bike, motorcycle) from the UDA-Part dataset, featuring detailed part annotations (e.g., wheels, doors).
- CLEVR: Used as the baseline for standard, low-complexity in-domain evaluation.
- GQA: A real-world VQA dataset used to validate that the question-redundancy insights discovered in Super-CLEVR successfully transfer to real-world data.
Key Insights and Ideas
- The Modular Trade-Off: Neural-modular and symbolic models (like NSCL and NSVQA) are highly robust against question redundancy shifts. Because they reason step-by-step, they don't easily exploit spurious linguistic shortcuts. However, they struggle with visual complexity because simple symbolic modules fail to capture intricate visual details.
- Non-Modular Strengths and Weaknesses: Dense, non-modular models (like FiLM and mDETR) handle visual complexity well due to their distributed representations, but their performance plummets when questions contain redundant descriptors or when concept distributions shift.
- Uncertainty Bridges the Gap: By injecting probabilistic uncertainty into a symbolic executor, P-NSVQA mitigates the primary weakness of neuro-symbolic models (visual vulnerability) while preserving their superior reasoning capabilities.
Key Results
- Benchmark Difficulty: Super-CLEVR proves to be significantly more challenging than the original CLEVR dataset. Even the state-of-the-art transformer model, mDETR, drops from 97.52% accuracy on CLEVR to 82.65% on Super-CLEVR.
- P-NSVQA Performance: P-NSVQA achieves the highest in-domain accuracy on Super-CLEVR at 95.76%, outperforming its deterministic baseline by nearly 3%.
- Out-of-Domain Robustness: To evaluate robustness, the authors measured Relative Degrade (RD)—the percentage of accuracy lost under a domain shift. P-NSVQA achieved the lowest degradation on three out of four factors, showing remarkable resilience to question redundancy (0.84% RD) and concept distribution shifts (13.72% RD).
Limitations and Future Work
The primary limitation is the synthetic nature of the Super-CLEVR dataset. While rendering 3D models provides perfect ground-truth control over domain factors, synthetic data cannot fully replicate the nuances of real-world imagery.
The authors note that future work should focus on translating these controlled diagnostic techniques into real-world datasets. Additionally, they suggest Super-CLEVR can be expanded to study hierarchical, part-based visual reasoning—an area where current VQA models still severely struggle.
Summary updated