Introduction
Compositional generalization is a fundamental hallmark of human intelligence, often described as the ability to understand and produce an infinite number of novel combinations from a finite set of known primitives. In the context of Artificial Intelligence, this "Lego-brick" logic implies that if a model understands the concept of a "red ball" and a "blue cube," it should inherently comprehend a "red cube," even if that specific combination was absent from its training data.
While significant progress has been made in single-modality compositional generalization, Vision-and-Language (V&L) tasks present a unique challenge. In Visual Question Answering (VQA), the primitives are distributed across two distinct modalities: the linguistic modality (words in a question) and the visual modality (objects and attributes in an image). This paper identifies and addresses Multi-Sourced Compositional Generalization (MSCG), a scenario where models must compose novel concepts by aligning primitives sourced from different modalities.
The MSCG Challenge
Existing research typically focuses on cases where the novel composition occurs within a single modality—for example, combining two known words in a new way. However, MSCG requires a model to bridge the gap between text and pixels. A model might encounter a linguistic attribute ("white") and must apply it to a visual entity (a specific dog seen in an image) to answer a question correctly.
To succeed, the model must not only understand the individual components but also achieve a precise semantic alignment between the word and its visual counterpart. Failure in MSCG often stems from a lack of consistent internal representations; if the model's concept of "white" in text does not perfectly map to its visual representation of "white" in an image, the composition fails.
Methodology: Retrieval-Augmented Training
To address the alignment gap, the authors propose a retrieval-augmented training framework. This approach is designed to help models learn consistent, unified representations for concepts regardless of their source modality. The framework consists of four primary components:
- Retrieval Database Construction: Two separate databases are curated from the training set. The Linguistic Database stores word-level primitives (e.g., "running," "dog," "blue"), while the Visual Database stores image patches representing objects and attributes. These databases capture the diversity of primitives across various contexts to prevent the model from overfitting to a single "prototype."
- Feature Retrieval: During the training phase, for every primitive identified in a sample, the system performs a search across both databases. Using cosine similarity, it retrieves the most semantically relevant examples from both the linguistic and visual domains.
- Feature Aggregation: The original feature of the primitive is "refined" by aggregating it with the retrieved features. This process forces the model to recognize that the word "dog" and a visual patch of a dog share a core semantic identity, effectively stabilizing the model's internal dictionary.
- Model Agnostic Integration: A key strength of this framework is its flexibility. It is designed as a "plug-and-play" module that can be integrated into diverse VQA architectures, ranging from specialized models like CFR to large-scale multimodal models such as Qwen-VL and LLaVA.
The GQA-MSCG Benchmark
To rigorously test this capability, the authors introduced GQA-MSCG, a benchmark derived from the standard GQA dataset. It contains 35,000 samples categorized by the nature of the novel composition:
- LL (Linguistic-Linguistic): Novel combinations of text-based primitives.
- VV (Visual-Visual): Novel combinations of image-based primitives.
- LV (Linguistic-Visual): The core MSCG challenge, requiring cross-modal composition.
The benchmark also introduces complexity "Levels" (1, 2, and 3) based on the number of novel compositions present in a single question-image pair.
Experimental Results
The retrieval-augmented framework demonstrated consistent performance gains across various model scales and architectures:
- Small-Scale Models: The CFR model saw its overall accuracy on GQA-MSCG improve from 72.41% to 74.21%.
- Large-Scale Multimodal Models: Qwen-VL (7B+ parameters) achieved a significant jump from 68.49% to 71.53%.
- Complexity Robustness: While performance naturally declines as the complexity level increases (from Level-1 to Level-3), the proposed framework significantly narrowed the gap compared to baseline models, showing greater resilience to complex reasoning tasks.
- General Performance: Importantly, the method does not penalize performance on standard, non-compositional tasks. On the VQA v2 benchmark, Qwen-VL improved from 69.04% to 69.82%, suggesting that better compositional logic enhances general visual reasoning.
Conclusion and Future Directions
This research highlights that MSCG is a distinct and critical hurdle in the path toward robust V&L models. By utilizing cross-modal retrieval, models can better align their internal concepts, leading to superior generalization in novel scenarios.
Future work may address the computational overhead associated with database retrieval during training and the reliance on upstream tools (like NLTK or Faster R-CNN) for primitive extraction. Additionally, scaling these retrieval databases to include massive, web-scale data could further empower models to handle truly "out-of-distribution" open-world scenarios.
Summary updated