Introduction
Generative Adversarial Networks (GANs) have achieved remarkable success in synthesizing high-fidelity images. However, standard GANs often operate on unstructured "black box" latent spaces, making it difficult to control specific scene elements. While recent "object-centric" models like BlockGAN introduced structured latent spaces to allow for individual object manipulation, they typically assume that objects are independent. This assumption fails in realistic scenarios where objects interact—colliding, stacking, or following traffic rules.
RELATE (RElational LATent space) addresses this by explicitly modeling spatial correlations and physical interactions between objects. By introducing a module that learns these physical constraints directly from raw, unlabeled data, RELATE enables the synthesis of physically plausible multi-object scenes and videos with state-of-the-art disentanglement.
Methodology
RELATE extends the object-centric GAN framework by introducing a dedicated Interaction Module ($\Gamma$) that learns physical constraints directly from raw images.
The generation process follows three primary stages:
- Independent Sampling: The model samples appearance vectors (defining object looks) and initial "provisional" pose vectors (defining initial locations) for each object.
- The Correction Step: The Interaction Module ($\Gamma$) processes all object parameters simultaneously. It acts as a correction network, adjusting provisional positions to ensure they are physically plausible—for example, preventing object interpenetration or ensuring balls stay within a container.
- Structured Rendering: These corrected parameters are mapped to 3D-aware feature tensors. A generator network ($G$) then synthesizes the final image from the combined background and foreground features.
To maintain interpretability, a position regressor ($P$) is employed to predict object locations from the generated output, forcing the generator to adhere to the provided spatial parameters. For video generation, RELATE incorporates a velocity-based update rule inspired by Neural Physics Engines, allowing it to model dynamic interactions over time.
Datasets and Evaluation
The model was evaluated across four diverse datasets:
- BALLSINBOWL: A synthetic environment with two colored balls rolling in an elliptical bowl.
- CLEVR: A standard benchmark for cluttered tabletop scenes with geometric primitives.
- ShapeStacks: A challenging dataset involving towers of stacked wooden blocks.
- REALTRAFFIC: A novel real-world dataset introduced in this work, consisting of five hours of intersection footage capturing complex vehicle interactions.
Results and Discussion
RELATE demonstrates significant improvements in both image quality and disentanglement compared to prior state-of-the-art models like BlockGAN and GENESIS.
Quantitative Performance
On the BALLSINBOWL dataset, RELATE achieved an FID score of 81.9, a substantial improvement over BlockGAN’s 152.3. This performance gain highlights a key insight: modeling correlations is mandatory for disentanglement. Without the interaction module, GANs often "cheat" by baking objects into the background to satisfy the discriminator, which degrades the model's ability to separate objects.
Scene Editing and Generalization
The structured latent space enables precise "point-and-click" scene editing, such as changing the color of a specific vehicle or moving objects within a stack while maintaining scene coherence. Notably, the model exhibits strong out-of-distribution generalization. For instance, after training on towers of 2–5 blocks, it successfully generated stable towers of 7 blocks, indicating it had learned underlying physical principles like gravity and balance without explicit supervision.
Conclusion and Future Work
RELATE proves that modeling object interactions is essential for effective scene disentanglement and realistic synthesis. By providing the appropriate architectural bias, GANs can learn complex physical concepts from raw data. Current limitations include a primary focus on planar (2D) motions and difficulties with drastic perspective changes in complex 3D scenes. Future research may explore more sophisticated 3D representations and explicit object segmentation to handle more complex, non-geometric shapes.
Summary updated