Exploring the Effect of Primitives for Compositional Generalization in Vision-and-Language

Chuanhao Li, Zhen Li, Chenchen Jing, Yunde Jia, Yuwei WuCVPR 2023 · OpenReview

Abstract

Compositionality is one of the fundamental properties of human cognition. Compositional generalization is critical to simulate the compositional capability of humans and has received much attention in the vision-and-language (V&L) community. It is essential to understand the effect of the primitives, including words, image regions, and video frames, to improve the compositional generalization capability. In this paper, we explore the effect of primitives for compositional generalization in V&L. Specifically, we present a self-supervised learning based framework that equips existing V&L methods with two characteristics: semantic equivariance and semantic invariance. With these two characteristics, the methods understand primitives by perceiving the effect of primitive changes on sample semantics and ground-truth. Experimental results on two tasks—temporal video grounding and visual question answering—demonstrate the effectiveness of our framework.

Introduction

While Vision-and-Language (V&L) models have achieved significant milestones in tasks like image captioning and video grounding, they often struggle with "compositionality"—the ability to understand novel combinations of familiar building blocks (primitives). For example, a model might correctly identify a "red cube" and a "blue sphere" during training but fail to recognize a "red sphere" in a test scenario.

Research indicates a startling "laziness" in current architectures: when 50% of critical words (like nouns or verbs) in a query are replaced with random ones, existing models still produce the same prediction nearly 90% of the time. This suggests that models over-rely on global context or dataset biases rather than truly parsing the relationship between individual primitives and the final semantics. This paper addresses this gap by introducing a self-supervised framework that teaches models to recognize the specific impact of words, image regions, and video frames.

Methodology

The proposed framework is model-agnostic and can be integrated into existing V&L architectures, such as 2D-TAN for video grounding or GLT for Visual Question Answering (VQA). The approach centers on two core concepts: semantic equivariance and semantic invariance.

Effect Estimation

To generate meaningful training samples, the framework first estimates the "importance" or effect of each primitive:

  • Words: Importance is ranked using NLTK (Nouns/Verbs > Adjectives > Articles).
  • Image Regions: CLIP is utilized to measure the similarity between specific image regions and query keywords.
  • Video Frames: High importance is assigned to frames within the ground-truth boundary, while visual entailment (via the TCL model) is used to evaluate others.

Sample Generation

Based on the estimated effects, the framework generates two types of augmented samples:

  1. Equivariant Samples: The model masks "high-effect" (critical) primitives. Because the core meaning is corrupted, the model is trained to change its prediction.
  2. Invariant Samples: The model masks "low-effect" (irrelevant) primitives, such as background pixels or articles like "the." The model is trained to maintain its original prediction.
Visualizing semantic equivariance and invariance through masking primitives.

Training Objectives

The framework employs a tripartite loss function to enforce these semantic properties:

  • Method-specific loss ($L_{ms}$): The standard loss for the underlying base model.
  • Self-supervised loss ($L_{ssl}$): Penalizes the model if it remains confident in the original answer when core semantics (equivariant samples) are masked.
  • Contrastive loss ($L_{cl}$): Pulls the representations of original and invariant samples together while pushing equivariant samples away in the embedding space.
The proposed self-supervised framework for temporal video grounding, illustrating the interaction between different loss components.

Experimental Evaluation

The framework was evaluated across two primary domains:

  • Temporal Video Grounding (TVG): Tested on Charades-CG (designed for compositional generalization) and Charades-STA. On the Charades-CG "Novel-Composition" split, the framework improved the MS-2D-TAN model's [email protected] from 43.17% to 46.54%.
  • Visual Question Answering (VQA): Evaluated on CLEVR and CLOSURE. Adding the framework to the GLT model resulted in a new state-of-the-art accuracy of 98.4% on CLOSURE, a 3.4% absolute gain over the baseline, without degrading standard performance.

Qualitative results show that the model becomes significantly more sensitive to semantic changes. For instance, in "Novel-Word" scenarios, the model can correctly identify rare objects because it has learned to attend to the specific primitives that define them rather than relying on frequent co-occurrence patterns.

Discussion and Conclusion

The core insight of this work is that a model's failure to generalize often stems from its insensitivity to semantic changes. While previous research focused heavily on invariance (robustness to noise), this paper demonstrates that equivariance—knowing when a change should matter—is equally critical for true compositional understanding.

Despite its success, the framework currently relies on pre-trained "teachers" like CLIP or NLTK for effect estimation. If these external models possess inherent biases, the generated self-supervised samples may be flawed. Future work will explore more "in-the-wild" datasets and aim to reduce the computational overhead associated with generating multiple masked versions of training samples.

Summary updated

Abstract

Compositionality is one of the fundamental properties of human cognition (Fodor & Pylyshyn, 1988). Compositional generalization is critical to simulate the compositional capability of humans, and has received much attention in the vision-and-language (V&L) community. It is essential to understand the effect of the primitives, including words, image regions, and video frames, to improve the compositional generalization capability. In this paper, we explore the effect of primitives for compositional generalization in V&L. Specifically, we present a self-supervised learning based framework that equips existing V&L methods with two characteristics: semantic equivariance and semantic invariance. With the two characteristics, the methods understand primitives by perceiving the effect of primitive changes on sample semantics and ground-truth. Experimental results on two tasks: temporal video grounding and visual question answering, demonstrate the effectiveness of our framework.
BibTeX
@inproceedings{li2023exploring,
  author = {Chuanhao Li and Zhen Li and Chenchen Jing and Yunde Jia and Yuwei Wu},
  title = {{Exploring the Effect of Primitives for Compositional Generalization in Vision-and-Language}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=lq7UubFpUiC},
}
Figure