Most systems that turn an image into a scene graph (a list of objects with boxes and labels, plus labeled relations such as "man wearing ring") run an object detector first and reason about relations second. If the detector misses the ring, no later stage can bring it back, and every relation involving it is lost. This note proposes to treat the detector's output as a rough first draft. A generative model then edits that draft toward the true graph in two to four steps. The edits move boxes, relabel objects and predicates, and, most importantly, switch on objects the detector never proposed, using spare "empty" node slots. The edit process mixes discrete jumps for labels and object existence with continuous motion for boxes. The draft already sits close to the answer, so the model only has to travel a short distance, which is why a few steps should suffice.
The technical problem is the step count. Fast one-step generators such as Mean Flows for One-step Generative Modeling work by predicting the average velocity over a time interval, and they require smooth continuous states. Labels and object existence are not smooth. Our observation is that the probability of a discrete jump over an interval is a smooth function of the interval's endpoints even though the state itself jumps. A model can therefore learn finite-interval jump probabilities for the discrete parts and average velocities for the box coordinates, both trained with the same derivative-based self-consistency. Put in two sentences for a colleague: start from the detector's graph instead of from noise, and learn a few large edit steps in which boxes slide and labels and "is this object here?" flags jump. The jumps are learned through their interval transition probabilities, which are differentiable in time, so one-step training tricks carry over to discrete variables.
Core Claim
A bridge from detector output to the true scene graph has two properties. First, it recovers a measurable fraction of objects that are completely absent from the detector's proposals, a failure that prior structured-vision systems concede they cannot fix. Second, at two to four network evaluations it matches or beats multi-step iterative refinement on tail-predicate recall, at a fraction of its latency. We also claim that modeling the discrete parts with learned interval jump kernels is what makes the few-step regime work, rather than encoding labels as continuous "analog bits" and rounding. Unlike the claim in the first paragraph, this last claim is a hypothesis with a clean ablation, and we treat it as falsifiable.
Where This Sits
The cascading-error problem is well documented. In GraphEQA, an agent's semantic memory is capped by what its detector sees. HostSG prunes noisy parser edges with an information bottleneck but states that never-detected entities cannot be resurrected. Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs and SG-Nav likewise filter or re-verify what was detected. These papers motivate the proposal. All of them can only remove or relabel nodes, whereas our model can add nodes.
Two scene-graph methods bracket the trade-off we target. Pixels to Graphs by Associative Embedding removes the detector stage by detecting objects and relations in one pass, but it reports 0% recall on rare predicates and collides in crowded scenes. Iterative Scene Graph Generation unrolls a Markov random field so that subject, object, and predicate guesses correct one another, roughly doubling tail recall, but its authors flag the latency of repeated decoding. Our method keeps IterativeSG's idea of mutual correction. It differs in three ways: the refinement is a learned generative transport with a small, fixed number of large steps; it starts from detector output rather than learned queries; and it has explicit slots for objects that were never proposed.
On the generative side, DiffuseSG generates grounded scene graphs with continuous diffusion over binary-bit label encodings plus boxes. It sidesteps the detector only because it is unconditional: it is not image-to-graph perception, and it samples with many steps. DiGress shows that native discrete jump noise on graphs trains faster than its continuous counterpart, reaching 99% QM9 validity in 1 hour versus 7.2 hours. FlowMol finds that a Gaussian relaxation of categories works on small molecules but collapses to 51.2% validity on GEOM-Drugs against 71.6% for MiDi. These results constrain our design and push us away from treating labels as continuous. MeanFlow and 3D MeanFlow supply the average-velocity method for the continuous coordinates. Discrete Adjoint Matching and the Wright–Fisher unification in A Unification of Discrete, Gaussian, and Simplicial Diffusion show that continuous-time jump processes (Markov chains that hop between categories at learned rates) can be manipulated with tools from the continuous side. We use that general lesson rather than their specific estimators.
Research Program
State. The state is a fixed-size set of node slots plus pairwise edge slots. Each node slot carries a continuous box and a discrete label drawn from the object classes plus an extra "absent" class. Each edge slot carries a predicate label, with "no relation" as one option. At time zero, detected objects fill their slots with the detector's boxes and top labels. A budget of extra slots starts "absent", with boxes drawn from a broad spatial prior. At time one, the slots hold the ground-truth graph, matched to the slots by bipartite matching as in IterativeSG. During training we build a conditional bridge between these endpoints. Boxes move along straight interpolation paths, and labels follow a jump process whose rates carry each slot from its starting label to its target label.
Model. A graph transformer reads the current state, the image features pooled at every slot's current box, and the interval endpoints $(r,t)$. It outputs two things: an average box velocity, as in MeanFlow, and, for each discrete slot, a transition matrix $K_\theta(r,t)$ whose entry $(i,j)$ is the probability of being in label $j$ at time $t$ given label $i$ at time $r$. This holds for the whole graph state at $r$, and slots are factorized given that state, as in standard discrete denoisers. Pooling image features at the moving boxes is the resurrection mechanism. An "absent" slot whose box drifts onto unexplained image evidence, or onto a location implied by a predicate on a neighbor, can jump to "present".
The discrete one-step identity. For a jump process with instantaneous rate matrix $Q(s)$ at time $s$, the interval kernel satisfies the forward equation
$$\frac{\partial K(r,t)}{\partial t} = K(r,t)\,Q(t), \qquad K(r,r)=I,$$
where $I$ is the identity matrix. This is the discrete analogue of MeanFlow's relation between average and instantaneous velocity. The bridge construction gives $Q(t)$ in closed form during training. $K_\theta$ outputs probabilities that vary smoothly in $t$, so the left side can be computed with the same Jacobian–vector product MeanFlow uses, at a similar modest overhead (MeanFlow reports about 16%). Training regresses the box average velocity with the MeanFlow loss and matches $K_\theta$ to the forward equation with a cross-entropy-style divergence. Supervision at $r=t$ anchors the instantaneous rates. We will state plainly that the factorization makes the joint kernel approximate. Measuring how much accuracy that approximation costs is part of the study.
Inference. We sample two to four large steps. Each step moves every box by its average velocity and samples every label from its interval kernel. All slots are updated in parallel.
Baselines
The naive reference is the frozen detector plus a standard relation head with no repair. Its omission-recovery rate is zero by construction, and it fixes the latency floor. A second naive reference is the same pipeline with a lowered confidence threshold and twice the proposals, tuned on validation for the best R@100. This tests whether omissions are really just low-scoring detections.
The strongest discriminative competitor is IterativeSG, retrained on our detector features and evaluated at its default number of refinement steps and at reduced step counts that match our network-evaluation budget. It is the strongest competitor because it is the only method in the file that corrects object errors using relation context, and it reports tail mR@100 of 15.8 against 7.1 for SGTR on Visual Genome. Beating it means matching or exceeding its tail mR@100 at equal measured wall-clock latency on the same GPU, or matching it at less than half its latency. The file has no reported overall SGGen R@100 for IterativeSG, so we reproduce it.
The single-pass reference is Pixels to Graphs by Associative Embedding, which reports SGGen Recall@100 of 11.3% on Visual Genome. That number is old, so we report it as a historical anchor and not as the bar.
The strongest generative competitor is an image-conditioned variant of DiffuseSG, which we must build ourselves because the file gives no image-conditioned numbers ("no reported result found on Lacuna for DiffuseSG on image-to-graph SGGen"). It starts from noise, uses the same transformer and image pooling, and is run with many steps and with distillation down to our step count.
Two ablations isolate the key ingredients. In the first, the bridge is unchanged but labels are binary-bit continuous variables trained with plain MeanFlow and rounded at the end. This removes the interval kernel. In the second, sampling starts from noise instead of from the detector's graph, which removes the warm start. All models share the detector, backbone, training images, a fixed GPU-hour budget, and an equal hyperparameter-search budget.
Why not just…?
Why not just lower the detector threshold or fine-tune the detector? This is the most serious threat. If most "omissions" are really proposals that score below the threshold, a recalibrated detector solves the problem for free. We will settle it directly. Each ground-truth object that the deployed pipeline misses is classified as either "present at some score among the top 1,000 proposals" or "absent from all proposals". We report recovery separately for each group and compare against the tuned low-threshold baseline. If our gains are confined to the first group, the resurrection claim fails and we will say so. The method would then reduce to a fast relation-aware rescorer.
Why not just run IterativeSG with fewer steps? Truncating an unrolled refinement network that was trained for many steps should hurt it, because it was never trained to make large jumps. But a version retrained for two steps might do fine. We include exactly that retrained, latency-matched variant. If it matches us on tail recall and recovery, our generative formulation adds nothing over discriminative refinement with spare queries.
Why not just apply MeanFlow to analog-bit labels, as DiffuseSG encodes them? It is simpler, and FlowMol shows that continuous relaxations can match principled discrete ones on small problems. FlowMol also shows them collapsing on larger, more complex graphs. Visual Genome's 150 object and 50 predicate classes over dense graphs sit closer to the hard regime. The first ablation in the Baselines section is this alternative, and it decides the question.
Why not just distill a multi-step bridge? Distillation needs a converged teacher and a staged training curriculum. MeanFlow's comparison against Shortcut-XL (FID 10.60) and iCT-XL (34.24) suggests that training directly for large steps does better in images. We include a distilled bridge at an equal total compute budget to check whether this carries over to graphs.
Proposed Experiments
The main benchmark is Visual Genome SGGen with Recall@K, mean Recall@K, and tail mean Recall@100, together with measured latency and network-evaluation counts. We add two new metrics: the recovery rate on ground-truth objects absent from all proposals, and the precision of resurrected nodes, so that recovery cannot be bought with hallucinations. Action Genome tests whether the same bridge repairs per-frame detections in video. IterativeSG reports a 12.6% mean Recall@20 gain over SimpleBase there, which gives a reference for refinement benefits. As a downstream stretch, we swap the bridge into GraphEQA's scene-graph construction on HM-EQA, where the reported success is 67.0% against 51.7% for Explore-EQA. An improvement there would show that recovered objects matter to an agent.
Three checks are decisive. First, recovery on the "absent from all proposals" group must be clearly above zero while resurrected-node precision stays comparable to the precision of detected nodes. Second, at two to four steps, tail mean Recall@100 must be at or above that of latency-matched, retrained IterativeSG. Third, the interval-kernel model must beat the analog-bit ablation at equal step count, and the gap should widen as steps decrease. If the gap vanishes, the discrete one-step identity is unnecessary and the contribution reduces to the warm-started bridge.
References
- Mean Flows for One-step Generative Modeling
- GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering
- Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labeling
- Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs
- SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation
- Pixels to Graphs by Associative Embedding
- Iterative Scene Graph Generation
- Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Models
- DiGress: Discrete Denoising diffusion for graph generation
- Mixed Continuous and Categorical Flow Matching for 3D De Novo Molecule Generation
- 3D MeanFlow: One-Step Point Cloud Completion and Generation via Average-Velocity Transport
- Discrete Adjoint Matching
- A Unification of Discrete, Gaussian, and Simplicial Diffusion