Generative Compositional Augmentations for Scene Graph Prediction

Boris Knyazev, Harm de Vries, Catalina Cangea, Graham W. Taylor, Aaron C. Courville, Eugene Belilovsky

TL;DR

Current scene graph prediction models crumble when encountering novel combinations of common objects and predicates (e.g., "cup on surfboard"). This paper introduces a generative data augmentation framework that perturbs existing scene graphs into plausible, rare compositions and uses a conditional GAN to synthesize their corresponding visual features. Training on these hallucinated feature augmentations improves zero-shot and few-shot relationship recognition without sacrificing overall scene graph prediction accuracy.

What Problem Does This Paper Address?

Scene Graph Generation (SGG) aims to parse an image into a structured graph of objects (nodes) connected by pairwise relationships (predicates/edges), forming triplets such as $\langle\text{person, on, surfboard}\rangle$. While foundational for vision-language tasks like Visual Question Answering and image captioning, SGG models struggle with compositional generalization.

Because visual relationship data is extremely long-tailed, models memorize spurious correlations from frequent compositions. At test time, when faced with zero-shot (ZS) triplets—compositions unseen during training—model accuracy plummets. Crucially, the authors show this is not purely a predicate imbalance issue: frequent predicates like "on" and common objects like "cup" dominate both the training set and the zero-shot test subset. Standard re-sampling or score-reweighting methods designed for class imbalance simply boost rare predicates (e.g., "walking on") at the expense of frequent ones, gutting all-shot recall without truly solving compositional reasoning.

Methods

Rather than capturing or hand-labeling expensive new data, the authors synthesize rare compositions by perturbing real scene graphs and generating synthetic visual features:

Real Graph (G) ──[ Perturbation ]──> Rare Graph (Ĝ)
                                           │
Precomputed Visual Features (V') ──────────┼──> [ GCN + Layout Refinement ] ──> Synthetic Features (Ĥ, V̂, Ê)
                                           │
Ground Truth Bounding Boxes (B) ───────────┘
  1. Scene Graph Perturbations: High-degree "hub" nodes in a ground-truth graph $G$ are perturbed to create a new graph $\hat{G}$:
  2. RAND: Replaces nodes with uniformly random categories (frequently implausible).
  3. NEIGH: Samples replacements from semantic neighbors in GloVe embedding space.
  4. GRAPHN (Graph-Structured Semantic Neighbors): Novel method that uses dataset co-occurrence statistics to find valid candidate replacements for neighboring edges. Candidates are weighted inversely by frequency, gated by a threshold parameter $\alpha$ (which trades off rarity against plausibility), and refined using top-$k$ semantic neighbors.
  5. Feature-Level Conditional GAN:
  6. Instead of rendering full RGB images, the generator directly synthesizes intermediate convolutional feature maps ($\hat{H}$), from which node ($\hat{V}$) and edge ($\hat{E}$) features are extracted via RoIAlign.
  7. To ease synthesis, the generator’s Graph Convolutional Network (GCN) concatenates graph representations with real pre-computed visual features ($V'$) of the target class, then uses layout refinement blocks conditioned on bounding boxes $\hat{B}$ (kept identical to ground-truth boxes $B$).
  8. Joint End-to-End Training:
  9. The SGG classification model $F$ is updated using both real data ($\mathcal{L}_{\text{CLS}}$) and synthetic data via a cycle-consistency reconstruction loss ($\mathcal{L}_{\text{REC}}$).
  10. Separate discriminators enforce marginal distributions for nodes ($D_{\text{node}}$) and edges ($D_{\text{edge}}$), while an unconditional global discriminator ($D_{\text{global}}$) regulates the joint feature map.
  11. Semantic Plausibility Metric: Uses a pretrained BERT model to score masked nodes in linearized triplets as a proxy metric for the plausibility of perturbed graphs.

Datasets

  • Visual Genome (VG): Evaluated on the widely used 150-object, 50-predicate benchmark split (57,723 training images, 26,446 test images).
  • Evaluation Subsets: To evaluate compositional generalization, the test set is split into zero-shot (triplets never seen in training; 4,519 images), 10-shot (1–10 training occurrences; 9,602 images), 100-shot (11–100 occurrences; 16,528 images), and all-shot settings.

Key Insights and Ideas

  • Compositional novelty differs from rare classes: Debiasing predicate frequencies does not fix compositionality. A triplet can be zero-shot even when every constituent element is in the top-5 most frequent classes (e.g., $\langle\text{cup, on, surfboard}\rangle$).
  • Structured perturbations beat language embeddings: Pure embedding similarity (NEIGH) produces combinations that are too safe and typical. GRAPHN’s combination of graph topology, dataset frequency inversion, and threshold filtering ($\alpha$) successfully explores the tail of plausible distributions.
  • The OOD generation tax: The GAN's visual feature fidelity drops significantly when conditioned on zero-shot graphs compared to seen test graphs (precision drops from 0.55 to 0.47). Generative models themselves struggle to generate out-of-distribution visual representations.

Key Results

  • Consistent Gains on Zero-Shot and All-Shot: On an enhanced Iterative Message Passing baseline (IMP++), GAN+GRAPHN improves Scene Graph Classification (SGCls) zero-shot recall from 9.27% to 9.89% while simultaneously raising all-shot recall from 48.70% to 50.06%.
  • Preserving Frequent Classes: Conventional re-weighting schemes boost zero-shot recall slightly (9.58%) but collapse all-shot recall down to 48.13% (and from 77.48% to 74.68% in predicate classification). The GAN augmentation avoids this penalty.
  • Model Agnostic: Applied to Neural Motifs (NM++), GAN+GRAPHN improves PredCls zero-shot recall by ~4% (jumping from 10.2% to 14.2% at R@50, and 13.4% to 17.4% at R@100).
  • Oracle Bounds: An "Oracle" perturbation using actual test-set ZS triplets reaches 10.11% zero-shot SGCls recall, showing that GRAPHN (9.89%) captures much of the available headroom under fixed bounding boxes.

Limitations and Future Work

  • Frozen Object Detector: The pipeline augments features extracted from a frozen Faster R-CNN detector. Because detector backbones cannot be fine-tuned on hallucinated visual features, improvements in end-to-end scene graph detection (SGGen/SGDet) remain modest.
  • Fixed Spatial Layouts: The model assumes perturbed entities share the original object's bounding box ($\hat{B} = B$). While autoregressively predicting new bounding boxes was attempted, the box-prediction model collapsed toward distribution means on zero-shot graphs, hurting SGG performance.
  • OOD Generative Degradation: Generating high-fidelity feature maps for genuinely novel compositions remains a bottleneck for the GAN generator. Addressing feature realism under extreme distribution shifts is an essential next step.

Summary updated

Abstract

Inferring objects and their relationships from an image in the form of a scene graph is useful in many applications at the intersection of vision and language. We consider a challenging problem of compositional generalization that emerges in this task due to a long tail data distribution. Current scene graph generation models are trained on a tiny fraction of the distribution corresponding to the most frequent compositions, e.g. <cup, on, table>. However, test images might contain zero- and few-shot compositions of objects and relationships, e.g. <cup, on, surfboard>. Despite each of the object categories and the predicate (e.g. ‘on’) being frequent in the training data, the models often fail to properly understand such unseen or rare compositions. To improve generalization, it is natural to attempt increasing the diversity of the training distribution. However, in the graph domain this is non-trivial. To that end, we propose a method to synthesize rare yet plausible scene graphs by perturbing real ones. We then propose and empirically study a model based on conditional generative adversarial networks (GANs) that allows us to generate visual features of perturbed scene graphs and learn from them in a joint fashion. When evaluated on the Visual Genome dataset, our approach yields marginal, but consistent improvements in zero- and few-shot metrics. We analyze the limitations of our approach indicating promising directions for future research.
BibTeX
@inproceedings{knyazev2021generative,
  author = {Boris Knyazev and Harm de Vries and Catalina Cangea and Graham W. Taylor and Aaron C. Courville and Eugene Belilovsky},
  title = {{Generative Compositional Augmentations for Scene Graph Prediction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=ZtxXFUAW6oH},
}

Loading related content...

Figure