Abstract
Humans have the ability to learn novel compositional concepts by recalling and generalizing primitive concepts acquired from past experiences. Inspired by this observation, in this paper, we propose MetaReVision, a retrieval-enhanced meta-learning model to address the visually grounded compositional concept learning problem. The proposed MetaReVision consists of a retrieval module and a meta-learning module which are designed to incorporate retrieved primitive concepts as a supporting set to meta-train vision-language models for grounded compositional concept recognition. Through meta-learning from episodes constructed by the retriever, MetaReVision learns a generic compositional representation that can be fast updated to recognize novel compositional concepts. We create CompCOCO and CompFlickr to benchmark the grounded compositional concept learning. Our experimental results show that MetaReVision outperforms other competitive baselines and the retrieval module plays an important role in this compositional learning process.
Introduction
Humans possess a remarkable ability for systematic generalization. If we understand the primitive concepts of "red" and "chair," we can identify a "red chair" even if we have never encountered that specific combination before. In contrast, modern Vision-Language Models (VLMs) like CLIP or LXMERT often struggle with such novel compositions, frequently relying on memorized patterns from training data rather than true compositional reasoning.
This paper introduces MetaReVision, a framework designed to bridge this gap in Grounded Compositional Concept Learning (GCCL). The core challenge of GCCL is the "domain-shift" between training on seen compositions and testing on novel ones. Since it is practically impossible to train a model on every possible combination of attributes and objects in the world, models must learn the underlying process of composition itself.
The MetaReVision Framework
MetaReVision tackles GCCL using a "Learn-Retrieval-Compose" strategy consisting of two primary components:
1. Retrieval Module
The system maintains an Element Concept Database. When presented with a query (e.g., an image of a red chair with the text "a [MASK] [MASK]"), the model uses a VLM-encoder and a FAISS indexer to retrieve the most relevant "primitive" examples from its memory—such as images of other red objects and other chairs. This acts as a form of external working memory, allowing the model to "remind" itself of what these primitives look like in different contexts.
2. Meta-Learning Module
Using Model-Agnostic Meta-Learning (MAML), the model treats these retrieved examples as a "support set." It performs a "fast-update" of its parameters to adapt to the specific primitives in the query before making a final prediction.
A key innovation here is the Verbalizer. In standard meta-learning, models can suffer from a "memorization problem" where they ignore the support set and guess based on global statistics. The Verbalizer forces the model to predict the query concept by selecting from the concepts present in the retrieved support set, ensuring the model actually utilizes the retrieved evidence.
Benchmarking Compositionality
To evaluate token-level compositional ability, the authors introduced two new benchmarks:
- CompCOCO: Derived from MSCOCO, featuring 24 specific "novel" pairs (e.g., "black bird" or "small dog") that are strictly excluded from the training set.
- CompFlickr: Derived from Flickr30k, following a similar construction to test zero-shot compositional acquisition.
These datasets are more granular than previous benchmarks, focusing on whether a model can accurately predict masked tokens for both attributes and objects in a compositional context.
Key Insights and Results
Superior Generalization
MetaReVision consistently outperformed "train-from-scratch" baselines and MAML models without retrieval. On the CompCOCO dataset, MetaReVision improved "Pair Accuracy" (correctly identifying both the attribute and the object) from approximately 7.7% to 13.5%. The framework proved effective across different VLM architectures, including both one-stream (VLBERT) and two-stream (LXMERT) models.
The Attribute Gap
One of the most striking findings is that attribute recognition (e.g., "red," "small") is significantly harder for models to generalize than object recognition (e.g., "dog," "bus"). Models are much better at identifying the "what" than the "how it looks," suggesting that attribute-object binding remains a major hurdle for current VLMs.
Diversity Matters
The authors found that the strategy for retrieval is critical. Retrieving diverse examples (the "Div 4" strategy) is more effective than simply picking the top 4 most similar ones. Diversity ensures the model sees different contexts for the same primitive, leading to more robust and flexible representations.
Conclusion and Future Work
MetaReVision demonstrates that combining retrieval with meta-learning provides a path toward more human-like compositional reasoning. However, the paper also highlights a massive performance gap: models achieve ~32% accuracy on seen compositions but drop to ~13% on novel ones.
Future improvements may involve explicit grounding, where textual concepts are specifically aligned to visual bounding boxes, and dynamic retrievers that can be trained end-to-end with the learner. As AI moves toward larger generative models, the challenge will be ensuring these models truly understand the logic of composition rather than simply memorizing the vast combinations found in web-scale data.
Summary updated