TL;DR
CoLa3D is a novel framework for 3D scene decomposition that operates directly within a shared, holistic latent space rather than generating objects independently. By extracting object-level meshes from a single whole-scene latent, it produces naturally aligned, physically consistent 3D assets that are $2\times$ faster to generate and significantly more accurate than previous state-of-the-art methods.
What Problem Does This Paper Address?
Creating "composable" 3D scenes—where a room is represented not just as a single mesh but as a collection of individual, interactable objects—is a holy grail for robotics, AR/VR, and digital twin creation.
The standard approach has been "segment-then-generate": identify objects in a 2D image, generate each object in its own "canonical" 3D space, and then try to "glue" them back together. This is notoriously fragile. Estimating the correct relative scale, pose, and position for these independently generated objects is difficult, often resulting in scenes where furniture floats in mid-air or clips through walls. Furthermore, these methods are computationally expensive, often requiring multiple diffusion model passes for a single scene.
Methods
CoLa3D shifts the paradigm from independent generation to joint decomposition within a shared latent space.
The core components include:
- Holistic Scene Latent: The model starts with a compact, structured 3D latent ($z^s$) representing the entire scene, generated from an image or an existing mesh using a foundation model like TRELLIS.2.
- Decomposition Transformer: This is a lightweight, query-based transformer. It takes the scene latent and a set of "queries" (which can be 2D masks, 3D points, or purely learnable slots) and learns to partition the scene latent into distinct object latents.
- Feature Enhancement: To help the model distinguish between objects, the geometric latent is augmented with 3D semantic features from Utonia and 2D visual cues from DINOv3.
- Parallel Decoding: Once the object latents are extracted, they are passed through a shared decoder in parallel. Because they were all carved out of the same global coordinate frame, the resulting meshes are naturally aligned without needing any post-hoc registration or pose estimation.
Datasets
The authors evaluated CoLa3D on a mix of synthetic and real-world benchmarks:
- Synthetic: 3D-FRONT (professionally designed furnished rooms) and MetaScenes (simulatable scenes derived from real-world scans).
- Real-World: Replica and ScanNet++, used for zero-shot generalization testing on challenging, scan-derived data.
- Scalability: The model was also trained on ARSG-110K, a large-scale dataset with over 110k scenes, to test its performance on denser environments with up to 20 objects.
Key Insights and Ideas
- The Power of Shared Coordinates: By performing decomposition in a shared latent space, spatial consistency is "baked in." The model naturally understands how objects relate to each other, drastically reducing physical interpenetration (clipping).
- Extreme Data Efficiency: Remarkably, CoLa3D outperforms SAM 3D while using less than 1% of its training data. This suggests that the latent decomposition architecture provides a much stronger inductive bias for scene understanding than simply scaling up per-object generation.
- Flexibility via Prompting: The query-based design allows the model to be "prompted" in various ways. You can give it a 2D mask from an image, click a 3D point in a scene, or let it automatically discover objects using learnable queries.
Key Results
CoLa3D achieves state-of-the-art performance across all tested benchmarks:
- Speed: It is roughly $2\times$ faster than existing feed-forward baselines like MIDI or SceneGen because it avoids redundant per-object processing.
- Accuracy: On the 3D-FRONT dataset, it reduced the scene-level Chamfer Distance (a measure of geometric error) by 27.9% compared to the best baseline. On real-world Replica data, the improvement was even more dramatic, with a 53.6% reduction in error.
- Physical Plausibility: Qualitative results show that CoLa3D handles heavy occlusion and close object contact much better than baselines, which often produce distorted shapes when objects overlap.
Limitations and Future Work
While a significant step forward, the authors note a few areas for improvement:
- Latent Resolution: The current compact latent grid can be too coarse to capture very thin objects (like lamp stands) or distinguish between objects that are packed extremely tightly together. A hierarchical latent structure is suggested as a potential fix.
- Data Bottleneck: Despite its efficiency, the model is still limited by the diversity of available 3D scene datasets, which are currently dominated by standard indoor furniture. Expanding to more diverse, "messy," or outdoor environments is a key next step.
- Safety and Ethics: As with any high-fidelity reconstruction tool, there are concerns regarding the potential for creating "deepfake" environments or reconstructing private spaces from casual photos without consent.
Summary updated