When several objects interact, the uncertainty about what happens next is mostly about the relations between them, not about each object on its own. Two billiard balls nearly tied for a collision, a block resting on the edge of another, two cars merging into one gap: in each case the open question is "does A hit B, and which way does B go?" Current object-centric world models are models that split a video frame into per-object "slots" and predict how each slot evolves. They make that prediction deterministically, or with one Gaussian (bell-curve) noise term per object. Models that do handle branching futures well place one flexible distribution over the whole scene. The leading example is the conditional flow variational autoencoder, a generative model whose prior is warped by a normalizing flow. A normalizing flow is a stack of invertible transformations that turns simple noise into a multi-peaked distribution with an exactly computable density. We propose to attach the stochastic latent variable, and its flow prior, to each pair of interacting objects instead.
The core claim is about where branching should be represented, not about flows for their own sake. Suppose $k$ interacting pairs each have two possible outcomes. A per-scene latent must then represent up to $2^k$ joint modes, and a per-object latent must coordinate its modes with every partner. An edge latent only ever has to represent two modes per pair. We therefore predict three things. Scene-level and object-level stochastic models should degrade as the number of simultaneously ambiguous interactions grows, while edge-level models should not. Edge-level models should carry their calibrated uncertainty over to scenes with more objects than they were trained on. Better-calibrated relational uncertainty should improve planning on relational manipulation tasks. The density family on the edge (flow, mixture, or categorical) is treated as a secondary choice and tested as an ablation.
Positioning Against Prior Work
The relational-reasoning literature supplies the deterministic backbone that we reuse.
- Interaction Networks predict a pairwise effect $e_{ij}$ for each pair of objects and update each object from the sum of its effects. They generalize zero-shot from 6-body to 12-body gravitational systems. We keep this structure and make $e_{ij}$ stochastic.
- Relation Networks compute a shared pairwise function over objects. The same paper places them downstream of a vanilla diagonal-Gaussian VAE, so all of its stochasticity lives in per-object Gaussian codes. That is exactly the design we argue is in the wrong place.
- Dual Attention Transformer and PrediNet show that explicit relation vectors beat implicit attention on relational benchmarks. Both are deterministic, so they motivate treating relations as first-class objects but say nothing about uncertainty over them.
On the generative side, three papers motivate the proposal by showing that flows fix unimodal collapse, but none places the flow on edges between perceived objects.
- CF-VAE shows that a context-conditioned flow prior stops Gaussian conditional VAEs from collapsing to straight-line mean trajectories on the Stanford Drone and HighD datasets. Its context is a single holistic vector.
- Set Flow builds a permutation-invariant flow over sets, but it routes all inter-object dependence through one pooled global vector.
- NP-FKGC uses flows to capture one-to-many relations, with large knowledge-graph gains (a 77.6% Hits@1 increase on NELL). It does this over symbolic triples, not over continuous perceived objects or dynamics.
SOLD supplies the perception and control setting. It uses slot-based dynamics from pixels and reaches 74.6% success on relational "Distinct" tasks, against 33.1% for DreamerV3. Its slot transitions carry no multi-modal relational latent.
The constraints we take seriously come from three sources. IAF needs gating tricks to stay stable. CF-VAE needs condition and posterior regularization to avoid collapse. Pairwise computation scales quadratically with the number of objects. We address these with discrete coupling flows rather than ODE-based flows such as E(n) flows, with CF-VAE's regularizers, and with sparse edges.
The Model
The model has three pieces: a stochastic edge latent, a deterministic slot update that uses it, and an ELBO training objective.
The edge latent. Each slot $s_i^t$ is the state of object $i$ at time $t$. For each candidate pair $(i,j)$, we keep the pair if the objects are within a learned proximity radius or among each slot's top-$m$ attention partners. Each kept pair gets one latent: $$z_{ij} = F_\theta(\epsilon_{ij};\, s_i^t, s_j^t), \qquad \epsilon_{ij} \sim \mathcal{N}(0, I).$$ Here $F_\theta$ is a short stack of conditional affine-coupling layers. The noise $\epsilon_{ij}$ is shared by the unordered pair, so both objects "agree" on what happened between them; for example, a collision happened for both or for neither.
The slot update. The next state is $$s_i^{t+1} = g_\psi\Big(s_i^t, \sum_{j} h_\psi(s_i^t, s_j^t, z_{ij})\Big),$$ where $h_\psi$ reads the shared latent in a direction-aware way. Training maximizes an evidence lower bound (ELBO). It uses an edge-level posterior $q(z_{ij}\mid s^t, s^{t+1})$, and the KL term is estimated with the flow's exact log-density. We apply CF-VAE's posterior-variance fixing and randomly withhold conditioning to keep the latent from being ignored.
Perception. Slots come from SAVi-style slot attention, as in SOLD. We first study the model on ground-truth object states, to separate the dynamics question from the brittleness of segmentation that SOLD reports.
Research Program
The work proceeds in three stages.
- Controlled physics. We use stochastic versions of the Interaction Network domains: bouncing balls and springs with sub-resolution perturbations that make specific collisions branch, plus blocks placed at tipping points. In these simulators the true branch probabilities are known, so mode coverage and calibration can be measured exactly. We vary $k$, the number of simultaneously ambiguous pairs, from 1 to 6. We train on scenes with 6 objects and test on scenes with 12.
- Real multi-agent data. We use HighD, where vehicle-to-vehicle yielding is the relational branch.
- Control from pixels. We use the SOLD manipulation suite plus a variant in which object contacts have randomized outcomes. Here the question is whether a planner that samples edge latents chooses actions that are robust to relational ambiguity.
Baselines
Deterministic reference. The naive reference point is a deterministic Interaction Network on states and deterministic SOLD on pixels. Both predict a single future. On the original deterministic balls domain the reported Interaction Network error is an MSE of 0.0020 (Interaction Networks). On our branching variants we expect it to average modes, and we report its minimum-of-K error with K=1 as a floor.
Strongest control competitor. This is SOLD (SOLD), which reports 74.6% average success on the Distinct tasks. We rerun it with the authors' recipe at the same environment-step budget, the same slot encoder and the same number of tuning trials as our model. Beating it means matching 74.6% within seed noise on the deterministic tasks, and exceeding it by a margin larger than the across-seed spread on the stochastic-contact variants.
Strongest density competitor. This is a CF-VAE-style global flow prior (CF-VAE). It is conditioned on the pooled or concatenated slots, uses the same coupling architecture and matched parameter count, and uses identical regularizers. CF-VAE reports a 20% improvement in negative conditional log-likelihood on MNIST strokes. It reports no absolute number on Stanford Drone or HighD; no reported result was found on Lacuna for HighD trajectory likelihood. We therefore rerun it ourselves on all three settings. Beating it means a better importance-weighted likelihood bound and better mode coverage at $k\ge 3$, and a gap that widens with $k$.
Ablations isolating the key ingredient. Each ablation changes one thing.
- Location of the latent. A per-slot flow prior moves the same flow from edges to objects. A per-slot diagonal Gaussian follows the Relation-Network-after-VAE design of Santoro et al..
- Density family. On the same edge placement, we compare a diagonal Gaussian, a Gaussian mixture and a categorical latent against the flow.
- Edge sharing. Separate $\epsilon_{ij}$ and $\epsilon_{ji}$ remove the shared unordered-pair noise.
All models get the same data, the same compute, and a 20-trial hyperparameter search each.
Why not just…?
Why not just put a flow prior over the whole scene, as CF-VAE does? This is the main threat. At small $k$ a global flow may perform just as well. Our argument is combinatorial: the number of joint modes grows as $2^k$, and nothing in the global model ties a mode to a pair, so it should not transfer from 6 to 12 objects. The sweep over $k$ and the entity-count transfer test settle this directly. If the global flow matches the edge flow at $k=6$ and at 12 objects, the central claim is wrong.
Why not just use a mixture or categorical latent instead of a flow? These may be enough when each relation has two or three clean outcomes. That is why the density family is an ablation and not the headline. We expect flows to help on continuous branches with skewed spread, such as deflection angles. If mixtures tie, we report that edge placement, not flows, is what matters.
Why not just make SOLD deterministic but bigger? A deterministic predictor trained with squared error converges to the mean of the branches. That is the same failure CF-VAE documents for Gaussian models on the Stanford Drone dataset. The deterministic baseline's minimum-of-K error and calibration on the known-probability physics tasks will show this directly. On deterministic tasks it should tie us, and we will report that tie.
Why not just use a standard transformer, whose attention already compares all pairs? DAT reports that standard transformers stay below 70% on match-pattern with 25,000 samples, while DAT reaches nearly 100% with 10,000. Implicit attention is therefore a weak relational representation even before stochasticity enters. Our per-slot flow baseline already includes transformer dynamics, so it covers this alternative.
Decisive Checks
Three outcomes decide the proposal.
- Combinatorial scaling. On the controlled physics tasks with $k=1$ to $6$, we compare branch coverage and the total-variation distance between predicted and true branch probabilities. If the edge-level advantage over the global and per-slot flows does not grow with $k$, the combinatorial argument fails.
- Uncertainty transfer. We compare the calibration of edge latents trained at 6 objects when tested at 12, using the transfer protocol from the Interaction Networks paper.
- Control. We compare success on the stochastic-contact SOLD variants against SOLD rerun at an equal budget. If calibrated relational uncertainty does not translate into planning gains there, the representation result stands but the claim that it matters for control does not.
References
- Interaction Networks for Learning about Objects, Relations and Physics
- Discovering objects and their relations from entangled scene representations
- Disentangling and Integrating Relational and Sensory Information in Transformer Architectures
- An Explicitly Relational Neural Network Architecture
- Conditional Flow Variational Autoencoders for Structured Sequence Prediction
- Set Flow: A Permutation Invariant Normalizing Flow
- Normalizing Flow-based Neural Process for Few-Shot Knowledge Graph Completion
- SOLD: Slot Object-Centric Latent Dynamics Models for Relational Manipulation Learning from Pixels
- Improving Variational Autoencoders with Inverse Autoregressive Flow
- E(n) Equivariant Normalizing Flows