RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning

Xiaojian Ma, Weili Nie, Zhiding Yu, Huaizu Jiang, Chaowei Xiao, Yuke Zhu, Song-Chun Zhu, Anima Anandkumar

TL;DR

RelViT enhances Vision Transformers (ViTs) for visual relational reasoning by introducing a "concept-feature dictionary" that guides the model during training. By using auxiliary tasks to align current image features with a "memory" of previously seen concepts, RelViT achieves massive gains in systematic generalization—outperforming baselines by up to 43% on challenging new benchmarks for HICO and GQA.

What Problem Does This Paper Address?

While deep learning has mastered basic image recognition, visual relational reasoning—understanding how entities in a scene interact (e.g., "a person riding a horse" vs. "a person feeding a horse")—remains a major hurdle.

The authors identify three core challenges:

  1. Object-centric learning: Identifying individual entities and their specific properties.
  2. Relational reasoning: Inferring the semantic links between pairs of entities.
  3. Systematic generalization: The ability to reason about novel combinations of known objects and relations (e.g., if a model knows "man riding bike" and "woman walking dog," can it understand "woman riding bike"?).

Current models often rely on "crutches" like pre-trained object detectors or rigid symbolic logic. While Vision Transformers (ViTs) are promising due to their built-in self-attention, they still struggle to generalize to unseen scenarios without explicit guidance.

Methods

The authors propose RelViT (Concept-guided Vision Transformer). The architecture is built on a standard ViT backbone but adds a clever training mechanism:

RelViT Architecture with Concept-Feature Dictionary
  • Concept-Feature Dictionary: This is a dynamic "memory bank" where the keys are semantic concepts (like "ride horse" or "table") and the values are queues of image features previously processed by the model.
  • Concept-Guided Global Task: During training, the model retrieves a feature from the dictionary that shares the same concept as the current image. It then tries to make the current image's representation similar to the retrieved one. This forces the model to cluster semantically related interactions together.
  • Concept-Guided Local Task: This task focuses on fine-grained details. It encourages the model to find "correspondences" between local patches in the current image and patches in a retrieved image of the same concept. This helps the model learn what specific parts of an image define an object or an interaction.
  • Self-Distillation: RelViT integrates these tasks into a self-supervised learning framework (similar to DINO or EsViT), where a "student" network learns from a "teacher" network (an exponential moving average of the student).

Datasets

The model was evaluated on two heavyweights of relational reasoning:

  • HICO (Human-Object Interaction): Focused on identifying how people interact with objects.
  • GQA (Visual Question Answering): Focused on answering complex, multi-step questions about a scene.

The "Systematic Split": A key contribution of this paper is the introduction of new "systematic splits" for these datasets. Instead of random testing, these splits specifically test the model on unseen combinations of objects and relations or longer reasoning chains (more "hops") than those seen during training.

Key Insights and Ideas

  • Cross-Image Reasoning: Most models look at one image at a time. By using the dictionary, RelViT effectively "looks back" at previous examples of a concept during every training step, providing a much stronger signal for what a "relationship" actually looks like.
  • ViTs as Object Detectors: The results suggest that ViTs, when trained with the right auxiliary tasks, can capture object-centric information so well that they can eliminate the need for external, pre-trained object detectors (like Faster R-CNN), which are computationally expensive and often act as a bottleneck.
  • The Power of "Delicate" Concepts: The authors found that using high-level interaction labels (like "riding a horse") as dictionary keys provided much better results than using simple object labels (like "horse"), proving that relational guidance is the secret sauce for generalization.

Key Results

RelViT demonstrated significant performance leaps over state-of-the-art baselines:

Performance comparison on HICO original and systematic splits
  • HICO: Improved mAP (mean Average Precision) by 16% on the original split and a staggering 43% on the new systematic split.
  • GQA: Improved accuracy by 13% on the original split and 18% on the systematic split.
  • Closing the Gap: On GQA, RelViT (using only raw pixels) achieved results nearly identical to models that used additional, expensive bounding-box features from pre-trained detectors.
  • Robustness: Ablation studies showed the method is "plug-and-play," working effectively across different ViT variants like Swin Transformers and PVTv2.

Limitations and Future Work

  • Label Dependency: While the concept-guided tasks are "auxiliary," they still require concept labels during the training phase. Exploring how to discover these concepts in a fully unsupervised manner is a natural next step.
  • Reasoning Complexity: While RelViT handles "hops" better than previous models, truly open-ended, human-level reasoning in highly cluttered or ambiguous real-world scenes is still an ongoing challenge.
  • Architectural Evolution: The authors suggest that while training objectives (like their auxiliary tasks) help immensely, future work could involve modifying the ViT architecture itself to better handle the "physics" of relationships.

Summary updated

Abstract

Reasoning about visual relationships is central to how humans interpret the visual world. This task remains challenging for current deep learning algorithms since it requires addressing three key technical problems jointly: 1) identifying object entities and their properties, 2) inferring semantic relations between pairs of entities, and 3) generalizing to novel object-relation combinations, i.e., systematic generalization. In this work, we use vision transformers (ViTs) as our base model for visual reasoning and make better use of concepts defined as object entities and their relations to improve the reasoning ability of ViTs. Specifically, we introduce a novel concept-feature dictionary to allow flexible image feature retrieval at training time with concept keys. This dictionary enables two new concept-guided auxiliary tasks: 1) a global task for promoting relational reasoning, and 2) a local task for facilitating semantic object-centric correspondence learning. To examine the systematic generalization of visual reasoning models, we introduce systematic splits for the standard HICO and GQA benchmarks. We show the resulting model, Concept-guided Vision Transformer (or RelViT for short) significantly outperforms prior approaches on HICO and GQA by 16% and 13% in the original split, and by 43% and 18% in the systematic split. Our ablation analyses also reveal our model's compatibility with multiple ViT variants and robustness to hyper-parameters.
BibTeX
@inproceedings{ma2022relvit,
  author = {Xiaojian Ma and Weili Nie and Zhiding Yu and Huaizu Jiang and Chaowei Xiao and Yuke Zhu and Song-Chun Zhu and Anima Anandkumar},
  title = {{RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=rAWK-pAZes},
}
Figure