Self-Supervised Relationship Probing

Jiuxiang Gu, Jason Kuen, Shafiq R. Joty, Jianfei Cai, Vlad I. Morariu, Handong Zhao, Tong Sun

TL;DR

Self-Supervised Relationship Probing (SSRP) is a novel framework that learns to identify visual relationships between objects in images without requiring expensive, human-labeled scene graph annotations. By leveraging the inherent structure of language (dependency trees) and contrastive learning, the model "probes" its own internal representations to discover implicit relationships, significantly improving performance on vision-language reasoning tasks.

What Problem Does This Paper Address?

Visual relationships—like "man riding bike" or "cup on table"—are the fundamental building blocks of scene understanding. Traditionally, models learn these relationships by training on datasets like Visual Genome, which contain human-annotated "scene graphs."

However, relying on supervised scene graphs has three major flaws:

  1. Annotation Bottleneck: Labeling every relationship in an image is incredibly expensive and time-consuming.
  2. The Long-Tail Problem: Human annotators tend to focus on common predicates (like "on" or "near"), leaving a "long tail" of rare but important relationships that models subsequently struggle to learn.
  3. Subjectivity: What one person calls "near," another might call "next to," leading to inconsistent training data.

The authors ask a compelling question: Can we learn these relationships implicitly by looking at how objects relate to each other in a model's representation space, using only raw images and captions?

Methods

The SSRP framework is designed to extract these implicit relationships using three main modules:

  • Intra-modality Encoders: BERT-like transformers that model relationships within a single modality (image-to-image or text-to-text).
  • Inter-modality Encoders: Modules that allow information to flow between visual and textual features via cross-modal attention.
  • Relationship Probe: This lightweight layer is the core innovation. It computes a distance matrix between all pairs of objects (or words) to find a projection where the distance between two entities represents the strength of their relationship.
Architectural variants of the SSRP framework showing intra- and inter-modality encoders alongside the relationship probes.

The Two-Stage Training Process

To avoid relying on human annotations, SSRP employs a clever two-stage training pipeline:

  1. Stage 1 (Pretraining): The model is trained on standard vision-language tasks, including Masked Language Modeling (MLM), Region of Interest (RoI) feature reconstruction, and Image-Text matching.
  2. Stage 2 (Probing): The backbone is frozen, and the Relationship Probes are trained.
  3. Text Side: The probe is trained to mimic "Gold" dependency trees (grammatical structures) generated by off-the-shelf NLP tools.
  4. Visual Side: Because there are no "gold" visual trees, the authors use Contrastive Learning. They assume that if an image is slightly augmented (flipped, rotated, or its bounding boxes jittered), the relationships between objects should remain stable. The model is trained to maximize the agreement of relationship structures across different views of the same scene.
The two-stage training pipeline utilizing contrastive learning to align visual and textual relationship structures.

Datasets and Augmentation

The authors primarily used MSCOCO, which contains 118k images and 591k image-caption pairs. To make the self-supervision effective, they expanded the dataset using horizontal flips, rotations, and bounding box jittering for images, alongside back-translation (English $\rightarrow$ German/Russian $\rightarrow$ English) for text. This resulted in roughly 1.65M visual samples and 1.77M sentence samples. The model was evaluated on NLVR2, VQA v2.0, GQA, and standard image captioning benchmarks.

Key Insights

  • Language as a Teacher: One of the most elegant ideas in SSRP is using the well-defined structure of language to guide the discovery of structure in images. Because the model learns a joint space, the linguistic dependency tree provides a "hint" for what the visual relationship graph should look like.
  • Stability is Structure: By enforcing that relationships must be invariant to image augmentations, the model learns to ignore noise and focus on the fundamental spatial and semantic links between objects.
  • Explicit Latent Variables: Unlike standard Transformers that hide relationships inside noisy "attention weights," SSRP uses an explicit distance matrix. This makes the learned relationships more interpretable for downstream tasks.

Key Results

  • Reasoning Superiority: On the NLVR2 dataset, the cross-modal variant ($SSRP_{Cross}$) achieved an accuracy of 75.71%, outperforming established models like LXMERT (74.9%) and VisualBERT (67.4%), despite being trained on significantly less data.
  • Efficiency: SSRP proved highly competitive on VQA and GQA tasks, matching the performance of models trained on datasets 10-15 times larger (e.g., LXMERT uses 9.2M images; SSRP uses 0.6M).
  • Qualitative Success: Visualizations of the "probed" trees showed that the model successfully identifies meaningful connections (e.g., linking a "person" to a "pizza" they are eating) without ever being explicitly taught the concept of "eating."

Limitations and Future Work

  • Predicate Blindness: While the model is excellent at identifying that a relationship exists and how strong it is, it doesn't explicitly name the relationship (e.g., it knows a man and a bike are connected, but doesn't output the specific word "riding").
  • Dependency on NLP Tools: The second stage of training relies heavily on the accuracy of the linguistic dependency parser. If the parser fails on complex sentences, the visual probe receives poor supervision.
  • Future Directions: The authors suggest this method could be used to "auto-label" massive unlabeled datasets, creating a virtuous cycle where self-supervised probes generate data to train even larger, more robust supervised models.

Summary updated

Abstract

Structured representations of images that model visual relationships are beneficial for many vision and vision-language applications. However, current human-annotated visual relationship datasets suffer from the long-tailed predicate distribution problem which limits the potential of visual relationship models. In this work, we introduce a self-supervised method that implicitly learns the visual relationships without relying on any ground-truth visual relationship annotations. Our method relies on 1) intra- and inter-modality encodings to respectively model relationships within each modality separately and jointly, and 2) relationship probing, which seeks to discover the graph structure within each modality. By leveraging masked language modeling, contrastive learning, and dependency tree distances for self-supervision, our method learns better object features as well as implicit visual relationships. We verify the effectiveness of our proposed method on various vision-language tasks that benefit from improved visual relationship understanding.
BibTeX
@inproceedings{gu2020self,
  author = {Jiuxiang Gu and Jason Kuen and Shafiq R. Joty and Jianfei Cai and Vlad I. Morariu and Handong Zhao and Tong Sun},
  title = {{Self-Supervised Relationship Probing}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure