Abstract
Compositional generalization—the ability to understand novel combinations of known primitives—is a cornerstone of human intelligence and a significant challenge for artificial intelligence. This paper introduces Multi-Sourced Compositional Generalization (MSCG) in the context of Visual Question Answering (VQA). Unlike traditional approaches that focus on single-modality novelties, MSCG explores unseen mixtures of linguistic and visual primitives. We present a retrieval-augmented training framework that aligns concepts across modalities by fetching semantically similar examples from dedicated text and image databases. Our approach significantly enhances the reasoning capabilities of both compact and large-scale VQA models, demonstrating robust performance on a newly introduced GQA-MSCG benchmark.
Introduction: The Challenge of Multi-Sourced Composition
Compositional generalization is often likened to building with Lego blocks: it is the capacity to take familiar components, such as "red," "dog," or "running," and interpret them in entirely new configurations, like a "running red dog," even if such a combination was absent from training data. In Visual Question Answering (VQA), this challenge is inherently multi-modal. A model must bridge the gap between linguistic primitives (words) and visual primitives (image patches).
The authors identify a critical bottleneck: models frequently struggle when these "blocks" originate from different sources. To generalize effectively, a model must not only recognize individual primitives but also align their representations so that the word "dog" and the visual depiction of a dog occupy a shared conceptual space.
Methodology: Retrieval-Augmented Training
To address the modality gap, the authors propose a model-agnostic retrieval-augmented training framework. Rather than learning primitives in isolation, the model is encouraged to leverage "semantic neighbors" during the training process.
The framework consists of several key stages:
- Primitive Databases: Two distinct databases are constructed from the training set: one for linguistic primitives (e.g., nouns, adjectives) and another for visual primitives (objects and attributes derived from scene graphs).
- Feature Retrieval: For every word in a query and every object in an image, the system retrieves the $K$ most semantically similar features from the respective databases using cosine similarity.
- Feature Aggregation: Original features are refined by averaging them with these retrieved neighbors. This representation-level data augmentation helps the model learn unified features for concepts, regardless of their modality.
- Integration: This "plug-and-play" module can be integrated into various VQA architectures, from small-scale models like CFR to large-scale vision-language models (LVLMs) like Qwen-VL.
The GQA-MSCG Benchmark
To evaluate this approach, the researchers introduced the GQA-MSCG benchmark, built upon the GQA dataset. This benchmark specifically targets novel compositions, categorized into three types:
- LL: Novel Linguistic-Linguistic pairs.
- VV: Novel Visual-Visual pairs.
- LV: Novel Linguistic-Visual pairs.
Test cases are further stratified into difficulty levels (Level-1 to Level-3) based on the number of novel composition types present in a single question, providing a granular assessment of model robustness.
Experimental Results
The retrieval-augmented (RAG) framework yielded consistent improvements across different model scales:
- Compact Models: The CFR model's overall accuracy on GQA-MSCG rose from 72.41% to 74.21%.
- Large-Scale Models: Qwen-VL experienced a significant boost, improving from 68.49% to 71.53%.
- General Performance: The method also enhanced performance on standard "Identically Distributed" (IID) datasets, with a 1.43% gain on GQA and a 0.78% gain on VQA v2, suggesting that better concept alignment benefits general VQA tasks.
Key Insights
The study reveals that the "modality gap" is a tangible hurdle; for instance, LLaVA-1.5 performed better on visual-visual (VV) compositions, while Qwen-VL excelled in linguistic-linguistic (LL) ones. By forcing unified representations through retrieval, models become more robust to novel contexts. Furthermore, the sharp decline in accuracy as composition "Levels" increase confirms that MSCG represents a distinct and more difficult challenge than standard compositional generalization.
Conclusion and Future Directions
While the retrieval-augmented framework shows great promise, it currently relies on scene graph annotations for visual primitive extraction. Future work should explore automated segmentation and labeling to remove this dependency. Additionally, as primitive databases scale, optimizing retrieval efficiency through advanced indexing will be crucial for maintaining training performance.
Summary updated