Abstract
Compositional generalization—the ability to understand novel combinations of known concepts—is a hallmark of human intelligence but remains a significant challenge for artificial intelligence. We present COVR (COmpositional Visual Reasoning), a new benchmark designed to evaluate this capability using real-world images. By leveraging an automated pipeline with human-in-the-loop validation, COVR introduces complex logical operations such as quantification and aggregation into visual question answering. Our evaluation of state-of-the-art vision-and-language models reveals a substantial performance gap between standard and compositional splits, with models often failing to generalize to unseen logical structures.
Introduction
Humans possess a remarkable capacity for systematicity: once we understand the concept of "all" and the category "dogs," we can immediately interpret the statement "all dogs are barking," even if we have never encountered that specific combination before. In contrast, current Visual Question Answering (VQA) models frequently rely on "reasoning shortcuts"—statistical correlations within training data—rather than a deep understanding of compositional logic.
While synthetic benchmarks like CLEVR have been instrumental in studying these issues, they utilize toy-like environments that do not reflect the visual complexity of the real world. Existing real-image datasets often lack the rigorous logical scaffolding required to test higher-order reasoning. COVR addresses this gap by providing a large-scale test-bed of real images paired with complex, compositionally-split logical queries.
Methodology
The COVR dataset was constructed using a multi-stage pipeline designed to ensure both logical complexity and visual diversity:
- Subgraph Extraction: The process begins with scene graphs from GQA, Visual Genome, and imSitu, which map objects and their relationships (e.g., man -> wearing -> glasses).
- Distractor Sampling: To prevent models from exploiting simple object-presence cues, the pipeline specifically selects "distracting" images. For a query about a "man wearing glasses," the set includes images of men without glasses or women wearing glasses.
- Template-Based Generation: 15 manually authored templates generate 640 distinct logical program types, covering operations like universal quantification ("all"), existential quantification ("some"), counting, and comparisons.
- Human-in-the-Loop Validation: To ensure linguistic naturalness and ground-truth accuracy, Amazon Mechanical Turk workers validated the generated questions and paraphrased them into fluent English.
- Compositional Splits: The dataset is partitioned into "Zero-shot" and "Few-shot" versions. Models are trained on individual concepts or simple structures and then tested on novel, more complex combinations of those same elements.
Dataset Characteristics
The resulting COVR dataset comprises 262,069 training examples across approximately 89,000 unique real-world images. The test set includes 13,900 manually validated and paraphrased questions. By building upon GQA, Visual Genome, and imSitu, COVR significantly expands the reasoning depth available for real-image benchmarks.
Key Insights
The Shortcut Problem
Models are adept at identifying "cheats." If a specific object class is highly correlated with a "True" answer in the training set, models often ignore the relational logic. COVR’s inclusion of hard distractors forces models to process the specific relationships described in the query.
Compositional vs. Lexical Generalization
The study distinguishes between learning new vocabulary (lexical) and learning new structures of meaning (compositional). While models handle new words relatively well, they struggle significantly when required to combine known logical constructs in new ways.
Structural Complexity and the "All" Gap
Generalization difficulty increases with the complexity of the logical structure. Models struggle particularly with universal quantification ("all"). While they may identify "some" instances of a property, they frequently fail to verify that a property holds across an entire set in a novel context.
Results and Discussion
The performance of state-of-the-art models like VisualBERT highlights a severe "reasoning gap." While achieving roughly 68% accuracy on standard i.i.d. data, performance drops precipitously on zero-shot compositional splits—reaching as low as 1.6% in specific cases.
A "Generalization Score" was developed to measure the gap between text-only baselines and upper-bound performance. In nearly half of the tested scenarios, this score remained alarmingly low (0%-70%). In contrast, human benchmarks reach approximately 92% accuracy, underscoring the limitations of current Transformer-based architectures in systematic reasoning.
Limitations and Future Work
The quality of COVR is inherently tied to the underlying scene graphs; approximately 70% of errors in the generated data stem from missing or incorrect annotations in the source datasets. Furthermore, while paraphrasing improves naturalness, the underlying logic remains template-bound. Future iterations may explore more diverse, human-generated logical structures to further challenge model architectures.
Summary updated