Abstract
Reasoning about visual relationships is central to how humans interpret the visual world. This task remains challenging for current deep learning algorithms since it requires addressing three key technical problems jointly: 1) identifying object entities and their properties, 2) inferring semantic relations between pairs of entities, and 3) generalizing to novel object-relation combinations, i.e., systematic generalization. In this work, we use vision transformers (ViTs) as our base model for visual reasoning and make better use of concepts defined as object entities and their relations to improve the reasoning ability of ViTs. Specifically, we introduce a novel concept-feature dictionary to allow flexible image feature retrieval at training time with concept keys. This dictionary enables two new concept-guided auxiliary tasks: a global task for promoting relational reasoning, and a local task for facilitating semantic object-centric correspondence learning. To examine the systematic generalization of visual reasoning models, we introduce systematic splits for the standard HICO and GQA benchmarks. We show the resulting model, Concept-guided Vision Transformer (or RelViT for short) significantly outperforms prior approaches on HICO and GQA by 16% and 13% in the original split, and by 43% and 18% in the systematic split. Our ablation analyses also reveal our model's compatibility with multiple ViT variants and robustness to hyper-parameters.
Introduction
While deep learning has mastered basic image recognition, "visual relational reasoning"—understanding how entities in a scene interact—remains a significant hurdle. For instance, a model might easily identify a "person" and a "horse," but distinguishing between a person riding a horse versus feeding a horse requires a deeper understanding of semantic links.
Current approaches to this problem face three main challenges: identifying individual entities (object-centric learning), inferring the semantic links between them (relational reasoning), and generalizing to novel combinations (systematic generalization). Historically, models relied on heavy-duty, pre-trained object detectors to provide "hints" about object locations. RelViT (Concept-guided Vision Transformer) argues that Vision Transformers (ViTs) possess the inherent potential to learn these relationships through self-attention, provided they are given a training signal that moves beyond simple pattern matching toward true relational understanding.
Methodology: Concept-Guided Learning
RelViT builds upon self-supervised learning frameworks like DINO and EsViT, introducing a "concept-guided" layer. The core innovation is the concept-feature dictionary ($D$), which acts as a form of long-term memory during training.
The Concept-Feature Dictionary
During training, the model maintains a memory bank (a queue) of image features indexed by "concepts" (labels such as "ride horse" or "apple"). This allows the model to compare current image features with those from previously seen images sharing the same semantic labels.
Auxiliary Reasoning Tasks
RelViT utilizes this dictionary to perform two new auxiliary tasks:
- Concept-Guided Global Task: When the model processes an image of a specific concept, it retrieves a feature of a different image with the same concept from the dictionary. The student network is then tasked with producing a representation similar to this retrieved memory. This encourages the model to cluster semantically similar interactions together in its latent space.
- Concept-Guided Local Task: To help the model learn exactly which parts of an image represent an object, RelViT performs patch-level matching. It compares patches from the current image to patches from a retrieved image of the same concept, encouraging the model to find semantic correspondences (e.g., matching the "horse" pixels in one image to the "horse" pixels in another) without needing explicit bounding box supervision.
Experimental Setup and Systematic Generalization
The authors evaluated RelViT on two major benchmarks:
- HICO (Human-Object Interaction): Focused on recognizing 600 different types of interactions between humans and objects.
- GQA (Visual Question Answering): Requiring multi-step relational reasoning to answer questions about a scene.
To measure true generalization rather than memorization, the authors introduced systematic splits for both datasets. Unlike standard random shuffles, these splits specifically test the model on "unseen" combinations of objects and relations (e.g., testing on "riding a horse" when the model has only seen "riding a bike" and "feeding a horse").
Key Results
RelViT demonstrated substantial performance leaps over both CNN-based models and standard ViTs:
- HICO Performance: RelViT improved the best baseline by 16% on the original split and 43% on the systematic "hard" split.
- GQA Performance: The model closed the performance gap between models using "cheating" features (from pre-trained object detectors) and those using only raw backbone features. It improved accuracy by 13% on the original split and 18% on the systematic split.
- Efficiency: Even with a relatively small backbone (PVTv2-b2 with 25.4M parameters), RelViT outperformed much larger ResNet-101 based models (44.7M parameters) that required extra supervision like bounding boxes or pose estimation.
Discussion and Insights
The success of RelViT suggests that the "image-as-patches" design of ViTs provides a natural inductive bias for object-centric representations, making them superior to CNNs for relational tasks. Ablation studies highlighted that the local task (patch matching) was particularly effective, suggesting that the primary bottleneck in visual reasoning is not the reasoning logic itself, but the ability to learn robust, object-centric features that remain consistent across different contexts.
Limitations and Future Work
While RelViT reduces the need for bounding boxes, it still requires concept labels during training to populate the dictionary. Future research could explore a purely unsupervised version of this dictionary. Additionally, while RelViT generalizes well to novel combinations, "productivity"—the ability to generalize to much longer reasoning chains than those seen in training—remains a difficult frontier for the field.
Summary updated