Closing the Loop: Graph-Level Latent Consistency for Scene-Graph-to-Image Generation

Image generators conditioned on a scene graph (a small diagram listing the objects in a picture, their attributes, and directed relations such as "dog riding motorcycle") still often return images that contradict the graph. They flip "A on B" into "B on A", drop objects, or change colors. The cause is that these generators run in one direction only. They map graph to image, and nothing checks whether the image, read back into a graph, says the same thing. Conditional image translation solved the analogous problem years ago with a latent consistency loss: an encoder must recover the conditioning code from the generated output, so a generator that ignores the code is penalized. That idea has never been applied in a form that respects the structure of a scene graph. A scene graph is an unordered set of nodes plus a set of directed, typed edges. It is not a fixed-length vector.

We propose to make scene-graph-conditioned diffusion generators close the loop. A scene-graph extractor reads each generated image. A matching step aligns the extracted objects to the input objects without assuming any ordering. The loss then penalizes disagreement node by node and edge by edge. Three details carry the idea. Only the graph part of the latent is required to round-trip, while texture and style are left free. The extractor is kept honest by continued supervision on real annotated images, so the generator cannot satisfy it with invisible tricks. Every claim is checked with an independent extractor that never touched training. The payoff we expect is concentrated in exactly the relations that box layouts cannot express, such as interactions ("holding", "riding") and attributes ("red").

Core Claim

A permutation-aware cycle loss over nodes and edges, applied to a structured part of the latent and paired with an extractor that stays grounded on real images, should raise how faithfully generated images preserve their input relations. The claim covers both full generation and localized editing, and it should hold without degrading image quality relative to the strongest feed-forward graph-conditioned diffusion model. We further predict where the gain appears. For purely geometric predicates ("left of", "above"), a box-level detector check should capture most of the benefit. For non-geometric predicates and attributes, only the graph-level cycle should help. If that split does not appear, the graph machinery is not earning its cost.

Relation to Prior Work

The method comes from the latent-consistency line. Toward Multimodal Image-to-Image Translation introduced the penalty $\|z - E(G(c,z))\|_1$ so that a translator cannot ignore its random code. Multimodal Unsupervised Image-to-Image Translation split the latent into content and style and reconstructed each separately. Towards Consistent Variational Auto-Encoding and Deep Automodulators showed that re-encoding a generator's own outputs stabilizes and disentangles the latent. All four treat the code as an ordered vector. We borrow the penalty and the content/style split, but replace the vector distance with a matched comparison over graphs.

The closest structured relative is Cycle Consistency Driven Object Discovery. It enforces slot-to-feature-to-slot and feature-to-slot-to-feature round trips, and raises ClevrTex foreground segmentation agreement (FG-ARI) from 0.62 to 0.72. That work motivates us, because it shows that cycles over unordered entity sets work. It also constrains us: its loss weights are reported to be highly sensitive. It has no relational edges and no generator conditioned on a graph, and it learns its slots without supervision. We add edges, put the cycle around a conditional generator, and anchor the extractor to labeled graphs.

BUSSARD is the only work in our file that brings exact bijectivity to scene graphs. It applies a normalizing flow to word embeddings of triplets in order to score anomalous relations. Its bijection never touches pixels, and its authors note that it cannot repair errors made by the upstream graph extractor. We move the round trip into the image-to-graph loop. Its finding that a flow needed a separate compression stage before it would train is one reason we use a soft cycle loss instead of exact invertibility.

The generators we would close the loop on are all open-loop. Image Generation From Scene Graphs uses a graph network, a box-and-mask layout, and refinement into 64×64 images trained with discriminators. Scene Graph Conditioning in Latent Diffusion injects graph embeddings into a pretrained diffusion model through ControlNet layouts or GLIGEN-style gated attention, and uses a contrastive graph encoder to separate inverse relations. CommonScenes generates 3D layouts and shapes from graphs. SGC-Net edits images by predicting regions from a modified graph and inpainting them with Stable Diffusion. Its authors report failures on attribute edits and small regions, which are exactly the failures a read-back check targets. None of these systems re-encodes its output.

Research Program

Latent structure. The latent has two parts. The graph part consists of node codes $z_i$, which carry category and attributes, and edge codes $r_{ij}$ for each ordered annotated pair, which carry the predicate. The residual part is a style code $w$ that carries texture, lighting, and anything else the graph does not specify.

Generator and extractor. The generator is a pretrained latent diffusion model with the gated-attention graph adapter of Scene Graph Conditioning in Latent Diffusion. The extractor $E$ is a scene-graph generation model that outputs object codes $\hat z_k$ and pairwise predicate distributions $\hat r_{kl}$.

The cycle loss. The training objective adds

$$\mathcal{L}_{\text{cyc}} = \min_{\pi} \Big[ \sum_{i} \ell_{\text{node}}(z_i, \hat z_{\pi(i)}) + \lambda \sum_{(i,j) \in \mathcal{E}} \ell_{\text{edge}}(r_{ij}, \hat r_{\pi(i)\pi(j)}) \Big].$$

Here $\pi$ is the assignment of input nodes to extracted objects, found by Hungarian matching on node cost and held fixed during backpropagation. $\ell_{\text{node}}$ and $\ell_{\text{edge}}$ are cross-entropies over categories, attributes, and predicates. $\mathcal{E}$ is the set of annotated directed edges, and $\lambda$ weights edges against nodes. Because the edge term indexes ordered pairs, swapping subject and object is penalized. The style code $w$ appears nowhere in the loss. This is the deliberate answer to the capacity problem raised in the latent-consistency literature: forcing pixels to round-trip a compact code would destroy texture, so only the structured part must survive.

Gradient path. Gradients reach the generator through the one-step predicted clean image, which is decoded and passed to $E$. This is applied only at low-noise timesteps, where that prediction is sharp.

Keeping the extractor honest. If $E$ is trained jointly with the generator, the two can collude by hiding graph information in imperceptible patterns. If $E$ is frozen, the generator can exploit its blind spots. We train $E$ on real Visual Genome or CLEVR annotations at the same time as it is used as a critic, and apply image augmentations before the critic sees generated images. We also never report fidelity with $E$ itself.

Reverse direction for editing. A second loop runs image → graph → edit → image → graph. It requires that the re-extracted graph equal the edited graph. Required equality covers both the targeted edge and every untargeted edge.

Missing annotations. Visual Genome graphs are incomplete. On real data we therefore penalize only annotated edges, never absent ones. On CLEVR, graphs are complete, so we additionally penalize hallucinated relations on non-edges.

Baselines

Naive reference. The naive reference point is the same pretrained diffusion model prompted with the graph flattened into text, with no adapter. This measures how much any structure buys over plain prompting.

Strongest existing method. The strongest competitor is the gated-attention graph-conditioned latent diffusion model of Scene Graph Conditioning in Latent Diffusion, including its contrastive hard-negative graph encoder. It reports an Inception Score of about 32.2 on COCO, against 22.2 for SceneGenie and 17.8 for SGDiff. It is the strongest because it already targets relation flips with hard negatives. Our method is a strict addition to it: same backbone, same adapter, same data, same number of fine-tuning steps, and the same hyperparameter search budget for both. We also give the baseline the extra steps our method spends, by continuing its standard training for the same wall-clock time. To beat it, we must improve relation recall, as measured by the held-out extractor and by humans, while keeping Inception Score and FID within noise of the baseline's own values under our evaluation pipeline. We found no reported number on Lacuna for relation-recovery accuracy of this baseline on generated images, so we will measure it ourselves.

Editing baseline. For editing we compare against SGC-Net. It reports an 8-point SSIM gain on edited regions in CLEVR and human preference of 9 to 33% over its baselines for removal and relation change.

Ablations. Four ablations isolate the key ingredient. The first replaces the graph cycle with a box-only cycle: a detector must recover the input boxes, but there is no edge term. The second freezes the extractor instead of keeping it grounded. The third removes the matching step and assumes a fixed node order. The fourth drops the style split, so that $w$ must also round-trip.

Why not just…?

Why not just use a pretrained scene-graph extractor as a frozen reward and fine-tune toward it? This is the most serious threat, because it is our method minus two pieces. Our concern is that real-image relation prediction is weak. Visual Genome reports baseline relationship accuracy near 8.7%, so a frozen critic gives noisy signals and has blind spots that a generator can exploit. The frozen-extractor ablation settles this directly. If it matches our method on the held-out extractor and in human judgments, the grounding component is unnecessary, and we will report it that way.

Why not just check boxes with an object detector, since most relations are spatial? For geometric predicates this may indeed suffice. That is why we split every fidelity result by predicate type. The proposal only stands if the graph cycle beats the box-only cycle on interaction predicates and attributes. Those are the categories SGC-Net reports failing on, and box layouts cannot encode them.

Why not just ask a vision-language model whether the image matches the graph? Generate Any Scene documents that large language models default to common patterns, such as putting a book on rather than under a table, and that they are roughly 90 times slower than programmatic graph processing for structured generation. GQA shows that strong models contradict themselves on related questions about 20% of the time. Such a model is too slow and too prior-biased to serve as a per-step training critic. We will still use one as one of several evaluation judges, alongside humans, so that our metric is not tied to our own extractor family.

Proposed Experiments

Controlled test on CLEVR. The first test uses CLEVR-style renders, where graphs are complete. An oracle extractor trained on abundant rendered data can be expected to read generated images reliably, given that Explainable and Explicit Visual Reasoning over Scene Graphs reaches near-perfect reasoning once graphs are available. We measure per-edge and per-attribute recovery, direction-flip rate, and hallucinated relations. We also use the CoGenT split, which holds out attribute combinations at test time. Its reported drop from about 85% to about 51% shows how fragile unseen combinations are, and graph-level cycles should narrow that gap.

Real-image test. The second test uses Visual Genome and COCO-Stuff, the benchmark used by Image Generation From Scene Graphs. Relation recall is measured by a held-out extractor from a different architecture family, by a vision-language judge, and by human raters on predicates stratified by type and by frequency. The long tail matters because rare predicates are where open-loop conditioning should fail first. Image quality is tracked by Inception Score and FID.

Editing test. The third test changes one edge or attribute and measures two things: whether the target change is recovered, and whether untargeted edges are preserved on re-extraction. Region SSIM is reported as in SGC-Net.

Decisive checks. The method succeeds if three results hold together. The graph cycle beats both the box-only cycle and the frozen-reward ablation on non-geometric predicates. That gain is confirmed by the held-out extractor and by humans, not only by the training critic. Image quality stays level with the baseline. A specific diagnostic guards against collusion. If gains appear only under the training extractor and vanish under the held-out one, the generator has learned to communicate with its critic rather than to draw the scene, and the grounding mechanism has failed.