COVR: A test-bed for Visually Grounded Compositional Generalization with real images

Ben Bogin, Shivanshu Gupta, Matt Gardner, Jonathan Berant

Abstract

While interest in models that generalize at test time to new compositions has risen in recent years, benchmarks in the visually-grounded domain have thus far been restricted to synthetic images. In this work, we propose COVR, a new test-bed for visually-grounded compositional generalization with real images. To create COVR, we use real images annotated with scene graphs, and propose an almost fully automatic procedure for generating question-answer pairs along with a set of context images. COVR focuses on questions that require complex reasoning, including higher-order operations such as quantification and aggregation. Due to the automatic generation process, COVR facilitates the creation of compositional splits, where models at test time need to generalize to new concepts and compositions in a zero- or few-shot setting. We construct compositional splits using COVR and demonstrate a myriad of cases where state-of-the-art pre-trained language-and-vision models struggle to compositionally generalize.

Introduction: The Challenge of Compositional Reasoning

Humans possess a remarkable ability for compositional generalization: once we understand the concept of "all" and the concept of "cheetahs," we can instantly understand the phrase "all cheetahs have spots," even if we've never encountered it before. In the field of Visual Question Answering (VQA), however, models often appear intelligent by exploiting "reasoning shortcuts" or statistical biases in datasets rather than truly understanding these compositions.

Existing benchmarks for testing true compositional reasoning have largely relied on synthetic images (like simple 3D shapes), which lack the complexity of the real world. Conversely, real-image datasets often lack the complex logical operators (like "all," "none," or "at least two") needed to test deep reasoning. COVR (COmpositional Visual Reasoning) bridges this gap by providing a large-scale test-bed of complex questions grounded in real-world photography.

The COVR Benchmark

The researchers developed an almost fully automatic pipeline to generate high-quality VQA pairs using real images from datasets like GQA, Visual Genome, and imSitu. The process involves:

  1. Subgraph Extraction: Extracting scenes from existing scene graphs (e.g., "a man catching a frisbee").
  2. Hard Distractor Sampling: Searching for "distractor" images that are visually similar but lack the specific relationship described, forcing the model to attend to visual details.
  3. Template-based Generation: Using a grammar of 15 templates to generate 640 unique logical programs, including higher-order operations like quantification and counting across multiple images.
  4. Automated Filtering and Human Validation: Using models like RoBERTa and LXMERT to ensure linguistic and visual consistency, followed by crowdsourced validation and paraphrasing of 13.9k examples into natural English.

The resulting dataset contains 262k examples based on approximately 89k real images.

Compositional Generalization and the "Hard Negative"

A key insight of COVR is the necessity of "hard negatives." VQA is only a true test of reasoning if the model is presented with challenging distractors. If a question asks about a "man wearing a hat" and only one image in the set contains a man, the model doesn't need to reason about the hat to succeed.

Examples of compositional splits in COVR, showing training on individual concepts and testing on novel combinations.

The benchmark specifically targets the gap between memorization and generalization. Models like VisualBERT achieve high accuracy on standard, identically distributed (i.i.d.) data but drop significantly on compositional splits. This suggests they are memorizing patterns rather than learning the underlying logic of quantifiers or relations. Generalization becomes exponentially harder as models are asked to combine more complex structures, such as processing "all women holding a purse AND sitting on a bench."

Key Results: The Generalization Gap

The experimental results reveal several critical weaknesses in current state-of-the-art models:

  • Performance Drop: In nearly half of the experimental setups, the generalization score was low (0%–70%), indicating a struggle to apply known concepts in new contexts.
  • Zero-Shot Failure: Models often failed completely on unseen templates in zero-shot settings, even when they had encountered all individual components of those templates during training.
  • Few-Shot Limitations: While models improved with a few examples, they required hundreds of instances to reach high accuracy—far more than the one or two examples typically needed by humans.
  • Human-Model Gap: Humans achieved 91.9% accuracy on the paraphrased test set, significantly outperforming models like VisualBERT and ViLBERT, which scored between 58% and 67%.

Conclusion and Future Directions

The COVR benchmark demonstrates that standard Transformer-based vision-language models are not "compositional by design." While the automatic generation process is limited by the quality of source scene graphs and template diversity, it provides a rigorous framework for evaluating deep reasoning. Future work should focus on developing new architectures capable of structural, symbolic-style reasoning to bridge the gap between machine pattern matching and human-like compositional understanding.

Summary updated

Abstract

While interest in models that generalize at test time to new compositions has risen in recent years, benchmarks in the visually-grounded domain have thus far been restricted to synthetic images. In this work, we propose COVR, a new test-bed for visually-grounded compositional generalization with real images. To create COVR, we use real images annotated with scene graphs, and propose an almost fully automatic procedure for generating question-answer pairs along with a set of context images. COVR focuses on questions that require complex reasoning, including higher-order operations such as quantification and aggregation. Due to the automatic generation process, COVR facilitates the creation of compositional splits, where models at test time need to generalize to new concepts and compositions in a zero- or few-shot setting. We construct compositional splits using COVR and demonstrate a myriad of cases where state-of-the-art pre-trained language-and-vision models struggle to compositionally generalize.
BibTeX
@inproceedings{bogin2021covr,
  author = {Ben Bogin and Shivanshu Gupta and Matt Gardner and Jonathan Berant},
  title = {{COVR: A test-bed for Visually Grounded Compositional Generalization with real images}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=OsZYkGMixb},
}
Figure