TL;DR
Current scene graph prediction models crumble when encountering novel combinations of common objects and predicates (e.g., "cup on surfboard"). This paper introduces a generative data augmentation framework that perturbs existing scene graphs into plausible, rare compositions and uses a conditional GAN to synthesize their corresponding visual features. Training on these hallucinated feature augmentations improves zero-shot and few-shot relationship recognition without sacrificing overall scene graph prediction accuracy.
What Problem Does This Paper Address?
Scene Graph Generation (SGG) aims to parse an image into a structured graph of objects (nodes) connected by pairwise relationships (predicates/edges), forming triplets such as $\langle\text{person, on, surfboard}\rangle$. While foundational for vision-language tasks like Visual Question Answering and image captioning, SGG models struggle with compositional generalization.
Because visual relationship data is extremely long-tailed, models memorize spurious correlations from frequent compositions. At test time, when faced with zero-shot (ZS) triplets—compositions unseen during training—model accuracy plummets. Crucially, the authors show this is not purely a predicate imbalance issue: frequent predicates like "on" and common objects like "cup" dominate both the training set and the zero-shot test subset. Standard re-sampling or score-reweighting methods designed for class imbalance simply boost rare predicates (e.g., "walking on") at the expense of frequent ones, gutting all-shot recall without truly solving compositional reasoning.
Methods
Rather than capturing or hand-labeling expensive new data, the authors synthesize rare compositions by perturbing real scene graphs and generating synthetic visual features:
Real Graph (G) ──[ Perturbation ]──> Rare Graph (Ĝ)
│
Precomputed Visual Features (V') ──────────┼──> [ GCN + Layout Refinement ] ──> Synthetic Features (Ĥ, V̂, Ê)
│
Ground Truth Bounding Boxes (B) ───────────┘
- Scene Graph Perturbations: High-degree "hub" nodes in a ground-truth graph $G$ are perturbed to create a new graph $\hat{G}$:
- RAND: Replaces nodes with uniformly random categories (frequently implausible).
- NEIGH: Samples replacements from semantic neighbors in GloVe embedding space.
- GRAPHN (Graph-Structured Semantic Neighbors): Novel method that uses dataset co-occurrence statistics to find valid candidate replacements for neighboring edges. Candidates are weighted inversely by frequency, gated by a threshold parameter $\alpha$ (which trades off rarity against plausibility), and refined using top-$k$ semantic neighbors.
- Feature-Level Conditional GAN:
- Instead of rendering full RGB images, the generator directly synthesizes intermediate convolutional feature maps ($\hat{H}$), from which node ($\hat{V}$) and edge ($\hat{E}$) features are extracted via RoIAlign.
- To ease synthesis, the generator’s Graph Convolutional Network (GCN) concatenates graph representations with real pre-computed visual features ($V'$) of the target class, then uses layout refinement blocks conditioned on bounding boxes $\hat{B}$ (kept identical to ground-truth boxes $B$).
- Joint End-to-End Training:
- The SGG classification model $F$ is updated using both real data ($\mathcal{L}_{\text{CLS}}$) and synthetic data via a cycle-consistency reconstruction loss ($\mathcal{L}_{\text{REC}}$).
- Separate discriminators enforce marginal distributions for nodes ($D_{\text{node}}$) and edges ($D_{\text{edge}}$), while an unconditional global discriminator ($D_{\text{global}}$) regulates the joint feature map.
- Semantic Plausibility Metric: Uses a pretrained BERT model to score masked nodes in linearized triplets as a proxy metric for the plausibility of perturbed graphs.
Datasets
- Visual Genome (VG): Evaluated on the widely used 150-object, 50-predicate benchmark split (57,723 training images, 26,446 test images).
- Evaluation Subsets: To evaluate compositional generalization, the test set is split into zero-shot (triplets never seen in training; 4,519 images), 10-shot (1–10 training occurrences; 9,602 images), 100-shot (11–100 occurrences; 16,528 images), and all-shot settings.
Key Insights and Ideas
- Compositional novelty differs from rare classes: Debiasing predicate frequencies does not fix compositionality. A triplet can be zero-shot even when every constituent element is in the top-5 most frequent classes (e.g., $\langle\text{cup, on, surfboard}\rangle$).
- Structured perturbations beat language embeddings: Pure embedding similarity (NEIGH) produces combinations that are too safe and typical. GRAPHN’s combination of graph topology, dataset frequency inversion, and threshold filtering ($\alpha$) successfully explores the tail of plausible distributions.
- The OOD generation tax: The GAN's visual feature fidelity drops significantly when conditioned on zero-shot graphs compared to seen test graphs (precision drops from 0.55 to 0.47). Generative models themselves struggle to generate out-of-distribution visual representations.
Key Results
- Consistent Gains on Zero-Shot and All-Shot: On an enhanced Iterative Message Passing baseline (IMP++), GAN+GRAPHN improves Scene Graph Classification (SGCls) zero-shot recall from 9.27% to 9.89% while simultaneously raising all-shot recall from 48.70% to 50.06%.
- Preserving Frequent Classes: Conventional re-weighting schemes boost zero-shot recall slightly (9.58%) but collapse all-shot recall down to 48.13% (and from 77.48% to 74.68% in predicate classification). The GAN augmentation avoids this penalty.
- Model Agnostic: Applied to Neural Motifs (NM++), GAN+GRAPHN improves PredCls zero-shot recall by ~4% (jumping from 10.2% to 14.2% at R@50, and 13.4% to 17.4% at R@100).
- Oracle Bounds: An "Oracle" perturbation using actual test-set ZS triplets reaches 10.11% zero-shot SGCls recall, showing that GRAPHN (9.89%) captures much of the available headroom under fixed bounding boxes.
Limitations and Future Work
- Frozen Object Detector: The pipeline augments features extracted from a frozen Faster R-CNN detector. Because detector backbones cannot be fine-tuned on hallucinated visual features, improvements in end-to-end scene graph detection (SGGen/SGDet) remain modest.
- Fixed Spatial Layouts: The model assumes perturbed entities share the original object's bounding box ($\hat{B} = B$). While autoregressively predicting new bounding boxes was attempted, the box-prediction model collapsed toward distribution means on zero-shot graphs, hurting SGG performance.
- OOD Generative Degradation: Generating high-fidelity feature maps for genuinely novel compositions remains a bottleneck for the GAN generator. Addressing feature realism under extreme distribution shifts is an essential next step.
Summary updated