TL;DR
RelViT enhances Vision Transformers (ViTs) for visual relational reasoning by introducing a "concept-feature dictionary" that guides the model during training. By using auxiliary tasks to align current image features with a "memory" of previously seen concepts, RelViT achieves massive gains in systematic generalization—outperforming baselines by up to 43% on challenging new benchmarks for HICO and GQA.
What Problem Does This Paper Address?
While deep learning has mastered basic image recognition, visual relational reasoning—understanding how entities in a scene interact (e.g., "a person riding a horse" vs. "a person feeding a horse")—remains a major hurdle.
The authors identify three core challenges:
- Object-centric learning: Identifying individual entities and their specific properties.
- Relational reasoning: Inferring the semantic links between pairs of entities.
- Systematic generalization: The ability to reason about novel combinations of known objects and relations (e.g., if a model knows "man riding bike" and "woman walking dog," can it understand "woman riding bike"?).
Current models often rely on "crutches" like pre-trained object detectors or rigid symbolic logic. While Vision Transformers (ViTs) are promising due to their built-in self-attention, they still struggle to generalize to unseen scenarios without explicit guidance.
Methods
The authors propose RelViT (Concept-guided Vision Transformer). The architecture is built on a standard ViT backbone but adds a clever training mechanism:
- Concept-Feature Dictionary: This is a dynamic "memory bank" where the keys are semantic concepts (like "ride horse" or "table") and the values are queues of image features previously processed by the model.
- Concept-Guided Global Task: During training, the model retrieves a feature from the dictionary that shares the same concept as the current image. It then tries to make the current image's representation similar to the retrieved one. This forces the model to cluster semantically related interactions together.
- Concept-Guided Local Task: This task focuses on fine-grained details. It encourages the model to find "correspondences" between local patches in the current image and patches in a retrieved image of the same concept. This helps the model learn what specific parts of an image define an object or an interaction.
- Self-Distillation: RelViT integrates these tasks into a self-supervised learning framework (similar to DINO or EsViT), where a "student" network learns from a "teacher" network (an exponential moving average of the student).
Datasets
The model was evaluated on two heavyweights of relational reasoning:
- HICO (Human-Object Interaction): Focused on identifying how people interact with objects.
- GQA (Visual Question Answering): Focused on answering complex, multi-step questions about a scene.
The "Systematic Split": A key contribution of this paper is the introduction of new "systematic splits" for these datasets. Instead of random testing, these splits specifically test the model on unseen combinations of objects and relations or longer reasoning chains (more "hops") than those seen during training.
Key Insights and Ideas
- Cross-Image Reasoning: Most models look at one image at a time. By using the dictionary, RelViT effectively "looks back" at previous examples of a concept during every training step, providing a much stronger signal for what a "relationship" actually looks like.
- ViTs as Object Detectors: The results suggest that ViTs, when trained with the right auxiliary tasks, can capture object-centric information so well that they can eliminate the need for external, pre-trained object detectors (like Faster R-CNN), which are computationally expensive and often act as a bottleneck.
- The Power of "Delicate" Concepts: The authors found that using high-level interaction labels (like "riding a horse") as dictionary keys provided much better results than using simple object labels (like "horse"), proving that relational guidance is the secret sauce for generalization.
Key Results
RelViT demonstrated significant performance leaps over state-of-the-art baselines:
- HICO: Improved mAP (mean Average Precision) by 16% on the original split and a staggering 43% on the new systematic split.
- GQA: Improved accuracy by 13% on the original split and 18% on the systematic split.
- Closing the Gap: On GQA, RelViT (using only raw pixels) achieved results nearly identical to models that used additional, expensive bounding-box features from pre-trained detectors.
- Robustness: Ablation studies showed the method is "plug-and-play," working effectively across different ViT variants like Swin Transformers and PVTv2.
Limitations and Future Work
- Label Dependency: While the concept-guided tasks are "auxiliary," they still require concept labels during the training phase. Exploring how to discover these concepts in a fully unsupervised manner is a natural next step.
- Reasoning Complexity: While RelViT handles "hops" better than previous models, truly open-ended, human-level reasoning in highly cluttered or ambiguous real-world scenes is still an ongoing challenge.
- Architectural Evolution: The authors suggest that while training objectives (like their auxiliary tasks) help immensely, future work could involve modifying the ViT architecture itself to better handle the "physics" of relationships.
Summary updated