Introduction
While generative models like StyleGAN have achieved remarkable success in creating realistic RGB images, they often fail to capture the underlying 3D geometry of a scene. This limitation is critical for applications requiring "complete" scene generation, where depth and surface orientation must align perfectly with the visual appearance. MultimodalGAN introduces a unified framework to generate consistent RGB images, depth maps, and surface normals from a single latent code, effectively learning the joint distribution of these modalities.
Methodology
The framework leverages the StyleGAN3 architecture as a shared backbone, modified with a branching structure to handle multiple modalities.
Shared Backbone and Branching
The generator uses a shared backbone for the initial layers to learn a universal scene representation. It then splits into separate branches for RGB, depth, and surface normals. The authors identified a "Goldilocks" branching point at Layer 9; branching too early prevents coherent structure, while branching too late introduces high-frequency artifacts.
Dual-Type Discriminators
To ensure both quality and alignment, the model employs two types of discriminators:
- Fidelity Discriminators (FD): Individual discriminators for each modality ensure that generated outputs (RGB, depth, or normals) are indistinguishable from real data of that type.
- Consistency Discriminator (CD): A joint discriminator evaluates the concatenated tuple of all modalities, forcing the generator to maintain strict geometric alignment across branches.
Training and Datasets
The model was trained on the Stanford2D3D dataset, which provides over 70,000 pairwise RGB-D-Normal samples from diverse indoor environments. To handle limited data, the authors utilized geometric translations for data augmentation, as standard color-based augmentations are incompatible with geometric modalities like depth.
Key Results
MultimodalGAN demonstrates significant improvements in both visual realism and geometric consistency. On the Stanford2D3D dataset, it achieved a Depth-FID of 24.8, a substantial leap over previous methods like DepthGAN (182.9).
GANs as Data Engines
A key contribution of this work is the use of synthetic data for downstream tasks. By augmenting real training sets with GAN-generated samples, the authors showed that a depth estimator (LeReS) trained on only 20% real data could match the performance of one trained on 80% real data. This highlights the potential of MultimodalGAN to serve as a high-fidelity data engine for computer vision.
Conclusion
By integrating a shared backbone with a consistency-focused adversarial loss, MultimodalGAN provides a robust solution for multimodal scene generation. While the model currently relies on statistical consistency rather than explicit 3D physics, its success suggests a promising path forward for scaling these principles to newer architectures like Diffusion Models and additional modalities such as semantic masks.