Abstract
Humans excel at learning novel compositional concepts by recalling primitives from past experience. We propose MetaReVision, a retrieval-enhanced meta-learning model designed for Grounded Compositional Concept Learning (GCCL). By incorporating retrieved primitive concepts as a support set to meta-train vision-language models, MetaReVision learns a generic representation that adapts rapidly to novel compositions. Evaluated on our new benchmarks, CompCOCO and CompFlickr, MetaReVision significantly outperforms competitive baselines.
Introduction: The Challenge of Compositional Generalization
Modern Vision-Language Models (VLMs) demonstrate impressive recognition capabilities but often fail at systematic generalization. While a human can easily identify a "blue apple" after seeing "red apples" and "blue cars," AI models tend to be "lazy" learners, memorizing specific co-occurrences seen during training. This failure in Grounded Compositional Concept Learning (GCCL) stems from a significant domain shift between seen and novel compositions that current models fail to bridge.
MetaReVision: Learn, Retrieve, and Compose
The MetaReVision framework addresses this by mimicking human cognitive processes through two primary modules:
- Retrieval Module: The system maintains an "Element Concept Database" indexed via FAISS. When presented with a query (e.g., a masked token in a caption), it retrieves diverse examples of the constituent "primitive" concepts (e.g., images of "red" objects and images of "chairs").
- Meta-Learning Module: Utilizing Model-Agnostic Meta-Learning (MAML), the model treats these retrieved primitives as a "support set." It performs a "fast update" to its parameters based on this context before predicting the final concept.
To ensure the model relies on the retrieved evidence rather than internal bias, a Verbalizer module constrains the output space to the concepts present in the support set, ensuring the model actually "looks" at the gathered evidence.
Benchmarking on CompCOCO and CompFlickr
To evaluate GCCL, researchers introduced two new benchmarks: CompCOCO and CompFlickr. These datasets feature a strict split where 24 specific attribute-object pairs (e.g., "blue bus") are held out from training and only appear during the "Novel Composition" testing phase.
Key Insights and Results
Experimental results on the LXMERT architecture demonstrate the effectiveness of this approach:
- Superior Generalization: MetaReVision achieved a Pair Accuracy of 13.79% on CompCOCO novel compositions, significantly outperforming the 8.14% baseline.
- Stability: The model improves on novel concepts without "forgetting" or degrading performance on seen compositions (maintaining ~34% accuracy).
- The Importance of Diversity: Retrieving a diverse set of examples (e.g., four different red objects) is more effective than retrieving the most similar ones, as redundancy in the support set limits the model's ability to abstract the primitive concept.
- Attribute Bottleneck: Analysis shows that while object recognition is robust, the primary challenge in GCCL lies in accurately grounding attributes (adjectives) across varied contexts.
Conclusion and Future Work
MetaReVision demonstrates that retrieval-augmented meta-learning is a powerful paradigm for teaching AI to "connect the dots" between visual concepts. Future work will explore explicit grounding mechanisms to align textual concepts with specific image regions and scaling these techniques to large-scale generative models.
Summary updated