Improving Predicate Representation in Scene Graph Generation by Self-Supervised Learning

So Hasegawa, Masayuki Hiromoto, Akira Nakagawa, Yuhei UmedaWACV 2023 · OpenReview

Abstract

Scene graph generation (SGG) aims to understand sophisticated visual information by detecting triplets of subject, object, and their relationship (predicate). Since the predicate labels are heavily imbalanced, existing supervised methods struggle to improve accuracy for the rare predicates due to insufficient labeled data. In this paper, we propose SePiR, a novel self-supervised learning method for SGG to improve the representation of rare predicates. We first train a relational encoder by contrastive learning without using predicate labels, and then fine-tune a predicate classifier with labeled data. To apply contrastive learning to SGG, we newly propose data augmentation in which subject-object pairs are augmented by replacing their visual features with those from other images having the same object labels. By such augmentation, we can increase the variation of the visual features while keeping the relationship between the objects. Comprehensive experimental results on the Visual Genome dataset show that the SGG performance of SePiR is comparable to the state-of-the-art, and especially with the limited labeled dataset, our method significantly outperforms the existing supervised methods. Moreover, SePiR’s improved representation enables the model architecture simpler, resulting in 3.6x and 6.3x reduction of the parameters and inference time from the existing method, independently.

Introduction

Scene Graph Generation (SGG) is a critical task in computer vision that involves transforming an image into a structured graph. In this graph, nodes represent detected objects (e.g., "boy," "skateboard"), and edges represent the predicates or relationships between them (e.g., "riding"). While object detection has matured significantly, SGG remains challenging due to the extreme class imbalance of predicates. In standard datasets like Visual Genome, common "head" predicates such as on or has appear with high frequency, whereas informative "tail" predicates like standing on or carrying are extremely rare.

Traditional supervised models often fail to recognize these rare categories, defaulting to the most frequent labels. While debiasing techniques exist, they often fail to provide the model with the necessary visual variety to learn robust representations of rare relationships. This paper introduces SePiR (Self-supervised learning for Predicate Representation), a framework designed to overcome these limitations by learning relational features without relying on predicate labels.

The SePiR Methodology

SePiR decouples the learning process into three distinct phases to ensure the model learns generalized relational features before being exposed to biased labels.

  1. Object Detection: The system utilizes an attention-based detector (DETR) to identify objects and extract their corresponding visual features.
  2. Self-Supervised Relational Encoding: This is the core of the SePiR framework. The model learns to represent relationships through contrastive learning. By using a novel data augmentation technique, the encoder learns to focus on the essential relational features that persist across different visual instances.
  3. Predicate Classification: Once the relational encoder is trained, it is frozen. A simple Multi-Layer Perceptron (MLP) is then trained as a classifier using the actual predicate labels.
The SePiR pipeline: (1) Object detection, (2) Self-supervised relational encoding using feature-swapping augmentation, and (3) Predicate classification.

Novel Data Augmentation: Feature Swapping

To facilitate contrastive learning, SePiR employs a unique augmentation strategy. It takes a subject-object pair and replaces their visual features with features of the same object types extracted from a different image. For example, the visual features of a "boy" and a "skateboard" in one image are swapped with those from another. This forces the relational encoder to learn a generalized "predicate-ness" that is independent of specific pixel arrangements or individual object appearances, focusing instead on the underlying relationship.

Experimental Results and Discussion

SePiR was evaluated on the Visual Genome (VG) dataset, including subsets with limited labels (5%, 10%, and 30%) to test robustness in data-scarce environments.

Performance in Data-Scarce Regimes

One of the most significant findings is SePiR's performance when labeled data is limited. With only 5% of labels available, SePiR outperformed state-of-the-art supervised methods by approximately 6 points in mean Recall (mR@100). This suggests that self-supervised pre-training provides a "cold start" advantage, allowing the model to understand the relational structure of the entire dataset before label bias can take hold.

Efficiency and Architecture

By improving the quality of the learned representations, SePiR allows for a much simpler model architecture. Unlike traditional SGG models that rely on heavy message-passing modules or complex Graph Neural Networks, SePiR achieves superior results with a basic MLP. This leads to a 3.6x reduction in parameters (93.6M vs. ~343M) and a 6.3x speedup in inference time compared to leading models like BGNN.

Evaluation on the PredCls task. The plot illustrates the trade-off between Recall@100 (head-heavy) and Mean Recall@100 (tail-aware), where SePiR demonstrates competitive performance across different regimes.

Tail Performance

Quantitative analysis confirms that SePiR is significantly more effective at identifying rare, informative predicates. While baseline models often default to generic predicates like on or in, SePiR more accurately identifies specific relationships such as parked on or flying in.

Conclusion

SePiR demonstrates that high-quality predicate representation learned through self-supervision is more valuable than architectural complexity in Scene Graph Generation. By leveraging feature-swapping augmentation, SePiR effectively addresses the long-tail problem, particularly in scenarios where labeled data is scarce. Future work may involve extending this approach to open-set images, utilizing the vast amount of unlabeled data available online to create even more robust foundation models for visual relationships.

Summary updated

Abstract

Scene graph generation (SGG) aims to understand sophisticated visual information by detecting triplets of subject, object, and their relationship (predicate). Since the predicate labels are heavily imbalanced, existing supervised methods struggle to improve accuracy for the rare predicates due to insufficient labeled data. In this paper, we propose SePiR, a novel self-supervised learning method for SGG to improve the representation of rare predicates. We first train a relational encoder by contrastive learning without using predicate labels, and then fine-tune a predicate classifier with labeled data. To apply contrastive learning to SGG, we newly propose data augmentation in which subject-object pairs are augmented by replacing their visual features with those from other images having the same object labels. By such augmentation, we can increase the variation of the visual features while keeping the relationship between the objects. Comprehensive experimental results on the Visual Genome dataset show that the SGG performance of SePiR is comparable to the state-of-theart, and especially with the limited labeled dataset, our method significantly outperforms the existing supervised methods. Moreover, SePiR’s improved representation enables the model architecture simpler, resulting in 3.6x and 6.3x reduction of the parameters and inference time from the existing method, independently.
BibTeX
@inproceedings{hasegawa2023improving,
  author = {So Hasegawa and Masayuki Hiromoto and Akira Nakagawa and Yuhei Umeda},
  title = {{Improving Predicate Representation in Scene Graph Generation by Self-Supervised Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=RctmolTWKI},
}
Figure