Generative Graph Perturbations for Scene Graph Prediction

Boris Knyazev, Harm de Vries, Catalina Cangea, Graham W. Taylor, Aaron C. Courville, Eugene Belilovsky

Generative Graph Perturbations for Scene Graph Prediction

Scene graph prediction involves identifying objects and their relationships within images. A significant challenge in this field is the "long-tail" distribution of relationships, where models struggle to generalize to rare or unseen "zero-shot" compositions. This paper proposes a generative data augmentation framework that uses Generative Adversarial Networks (GANs) to "hallucinate" rare training examples. By perturbing existing scene graphs and leveraging BERT to ensure these new combinations remain plausible, the method aims to improve model performance on unseen triplets.

The Long-Tail Challenge in Scene Graphs

Standard scene graph models often rely on statistical correlations rather than learning underlying compositional logic. For instance, a model might frequently encounter "person riding horse" but rarely "person riding giraffe." Consequently, performance drops significantly when encountering unseen compositions.

The long-tail distribution of triplets in training data leads to a sharp decline in recall for zero-shot scenarios.

As shown in the frequency distribution, recall at 100 (R@100) falls from 30% for frequent triplets to just 4.5% for unseen ones. The authors argue that to overcome this, models must be exposed to a greater diversity of compositions during training without the high cost of manual annotation.

Proposed Methodology: Generative Data Augmentation

The authors introduce a pipeline that generates visual features ($V$ for objects and $E$ for relationships) rather than raw pixels, which is more computationally efficient and avoids the artifacts of image generation.

The proposed architecture features a Conditional GAN (CGAN) that processes perturbed scene graphs to generate synthetic visual features.

1. Conditional GAN (CGAN)

The model uses a Graph Convolutional Network (GCN) to process input scene graphs, mapping them to a spatial layout. Convolutional layers then produce feature maps representing the scene.

2. Graph Perturbations

To create new training samples, the system "perturbs" real scene graphs:

  • Random Perturbation: Labels are changed randomly.
  • BERT-based Perturbation: A pretrained BERT model suggests plausible replacements (e.g., "person riding [MASK]" might become "person riding bike"), injecting world knowledge into the process.

3. Multi-level Discriminators and Joint Training

Three discriminators (node, edge, and global) ensure the generated features are realistic. The generative model is trained jointly with the scene graph classifier, allowing the latter to learn from both real and synthetic data.

Experimental Setup and Results

The framework was evaluated on the Visual Genome dataset, focusing on the 150 most frequent object classes and 50 predicate classes. The results indicate modest improvements in generalization:

  • Zero-Shot Recall ($R_{ZS}$): The baseline recall of 9.1% improved to 9.6% with random perturbations and 9.5% with BERT-based contextual perturbations.
  • Predicate Classification: Zero-shot recall rose from 28.4% to 29.3% using BERT with context.

Ablation studies confirmed that the spatial layout and specific loss functions are critical to the pipeline's performance.

Discussion and Limitations

While the gains were incremental, the study highlights key hurdles in scene graph prediction. Sparse connections in Visual Genome limit the context available for BERT to suggest meaningful perturbations. Additionally, the limited class split may not fully utilize the linguistic power of models like BERT. Finally, generating plausible spatial layouts for novel, out-of-distribution scenes remains a significant technical challenge. Future work may focus on more robust "layout-to-feature" models capable of handling the complexities of novel compositions.

Summary updated

BibTeX
@inproceedings{knyazev2021generative,
  author = {Boris Knyazev and Harm de Vries and Catalina Cangea and Graham W. Taylor and Aaron C. Courville and Eugene Belilovsky},
  title = {{Generative Graph Perturbations for Scene Graph Prediction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=ZtxXFUAW6oH},
}
Figure