Tackling the Unannotated: Scene Graph Generation with Bias-Reduced Models

Tzu-Jui Julius Wang, Selen Pehlivan, Jorma Laaksonen

TL;DR

This paper tackles the challenge of generating scene graphs (visual relationships between objects in an image) from unannotated data, which is a significant problem in current datasets. They propose a novel two-learner framework that uses a "skeptical" learner to guide a "supervised" learner, effectively reducing biases from incomplete annotations and improving the detection of rare relationships.

What Problem Does This Paper Address?

Scene graph generation (SGG) aims to represent an image's content as a graph where nodes are objects and edges are their relationships. While SGG models have advanced, their performance is hampered by the Visual Genome (VG) dataset, which suffers from two main issues:

  1. Annotation Bias: VG annotations are heavily skewed towards frequent relationships (e.g., "on," "has," "near"), making it easy for models to achieve high overall recall by simply predicting these common relations. This leads to a poor understanding of less frequent but equally important relationships.
  2. Sparse and Incomplete Annotations: Many object pairs in images are not annotated at all (false negatives), or they have multiple valid relationships that are only partially captured (partially true positives). Existing SGG models often overlook these issues, leading to suppressed likelihoods of correct predictions or failure to learn multi-modal relationships.

This paper addresses these biases to improve the accuracy and robustness of SGG models, particularly for underrepresented relationships.

Methods

The core of the proposed method is a novel SGG framework featuring a two-learner approach for relation classification, inspired by knowledge distillation and co-training.

  • Main Learner (F): This is a standard supervised learner that trains on all relation classes, including treating unannotated pairs as "no-relation."
  • Skeptical Learner (G): This auxiliary learner is trained on all relations except "no-relation." It makes no assumption that unannotated pairs are unrelated, encouraging it to detect as many potential relations as possible.

The key innovation lies in how these learners interact:

  1. Knowledge Distillation (KD): Learner G transfers its "less biased" knowledge (predictions) to learner F. This regularization helps F overcome the biases present in the dataset's annotations.
  2. Two KD Variants:
    • Uniform KD (uKD): A simple approach where knowledge is transferred uniformly.
  3. Certainty-based KD (cKD): A more sophisticated approach where F is forced to absorb more knowledge from G's more confident predictions. This is achieved by weighting the knowledge transfer based on the entropy of G's predictions.
  4. Hyperspherical Space: The models are trained in a hyperspherical space, which is shown to be better at capturing minority class distributions.
  5. L2-Normalization: Input representations and classifier weights are L2-normalized before calculating logits, a technique known to improve performance on low-data regimes and minority classes.

This framework is designed to be compatible with existing SGG models.

Datasets

The primary dataset used for training and evaluation is the Visual Genome (VG) dataset. The authors follow standard splits for training, validation, and testing. They consider the most frequent 150 entity classes and 51 relation classes (including "no-relation").

Key Insights and Ideas

  • The "No-Relation" Class is Artificial: The paper argues that the "no-relation" class, often used to handle unannotated pairs, is an artifact of incomplete annotations and can introduce bias. Learner G's ability to predict relations even for unannotated pairs directly challenges this assumption.
  • Skeptical Learner as a Bias Corrector: By not assuming "no-relation," learner G acts as a crucial source of regularization for learner F, pushing it to consider a wider range of relationships and reducing its reliance on biased priors.
  • Certainty-Based Knowledge Distillation is Key: The cKD variant, which prioritizes knowledge from G's more confident predictions, proves more effective than uniform KD, leading to more balanced improvements in recall and mean class-wise recall.
  • Hyperspherical Space for Minority Classes: Training in a hyperspherical space demonstrably helps in capturing the distribution of less frequent relation classes.

Key Results

The proposed framework, particularly with the L2+cKD configuration, shows significant improvements:

  • Mean Class-wise Recall (mR@K): Achieved substantial relative improvements across all tasks (SGDet, SGCls, PrdCls), ranging from +6.6% to +20.4%. This is the primary metric targeted, indicating better performance on rare relationships.
  • Overall Recall (R@K): Maintained competitive or slightly improved recall, with changes ranging from -2.4% to +0.3%. This demonstrates that the gains in mR@K do not come at a significant cost to overall performance.
  • Comparison to Baselines: The models consistently outperform their respective baselines (Motifs and VCTree) on mR@K, with improvements like +7.18% for VCTree L2+cKD in SGDet with constraint.
  • Class-wise Analysis: Figure 2(b) visually confirms that the proposed method significantly improves recall for minority classes (e.g., "under," "wearing," "in front of") compared to the baseline.

Limitations and Future Work

  • Dataset Bias: While the method significantly reduces bias, the underlying VG dataset still contains inherent limitations. The paper acknowledges that the improvements are achieved by better handling these biases.
  • Generalization to Rare/Unseen Triplets: The authors suggest that a future direction is to investigate the generalization ability of their models on rarely seen (few-shot) or entirely unseen (zero-shot) subject-relation-object triplets. This would push the boundaries of SGG beyond the annotated relationships in the dataset.
  • Temperature Hyperparameter: An ablation study (Table 2) shows that a higher temperature T can improve mR@K but at the cost of a deeper drop in R@K, suggesting a trade-off that needs careful tuning.

Summary updated

Abstract

Predicting a scene graph that captures visual entities and their interactions in an image has been considered a crucial step towards full scene comprehension. Recent scene graph generation (SGG) models have shown their capability of capturing the most frequent relations among visual entities. However, the state-of-the-art results are still far from satisfactory, e.g. models can obtain 31% in overall recall R@100, whereas the likewise important mean class-wise recall mR@100 is only around 8% on Visual Genome (VG). The discrepancy between R and mR results urges to shift the focus from pursuing a high R to a high mR with a still competitive R. We suspect that the observed discrepancy stems from both the annotation bias and sparse annotations in VG, in which many visual entity pairs are either not annotated at all or only with a single relation when multiple ones could be valid. To address this particular issue, we propose a novel SGG training scheme that capitalizes on self-learned knowledge. It involves two relation classifiers, one offering a less biased setting for the other to base on. The proposed scheme can be applied to most of the existing SGG models and is straightforward to implement. We observe significant relative improvements in mR (between +6.6% and +20.4%) and competitive or better R (between -2.4% and 0.3%) across all standard SGG tasks.
BibTeX
@inproceedings{wang2020tackling,
  author = {Tzu-Jui Julius Wang and Selen Pehlivan and Jorma Laaksonen},
  title = {{Tackling the Unannotated: Scene Graph Generation with Bias-Reduced Models}},
  booktitle = {BMVC},
  year = {2020},
  url = {https://openreview.net/forum?id=aOWwkxT5Wi},
}
Figure