CoLa3D: Composable Latent 3D Decomposition

Xudong Cai, Yongcai Wang, Deying Li, Chuanxia ZhengarXiv 2026 · arXiv

TL;DR

CoLa3D is a novel framework for 3D scene decomposition that operates directly within a shared, holistic latent space rather than generating objects independently. By extracting object-level meshes from a single whole-scene latent, it produces naturally aligned, physically consistent 3D assets that are $2\times$ faster to generate and significantly more accurate than previous state-of-the-art methods.

What Problem Does This Paper Address?

Creating "composable" 3D scenes—where a room is represented not just as a single mesh but as a collection of individual, interactable objects—is a holy grail for robotics, AR/VR, and digital twin creation.

The standard approach has been "segment-then-generate": identify objects in a 2D image, generate each object in its own "canonical" 3D space, and then try to "glue" them back together. This is notoriously fragile. Estimating the correct relative scale, pose, and position for these independently generated objects is difficult, often resulting in scenes where furniture floats in mid-air or clips through walls. Furthermore, these methods are computationally expensive, often requiring multiple diffusion model passes for a single scene.

Methods

CoLa3D shifts the paradigm from independent generation to joint decomposition within a shared latent space.

Overview of the CoLa3D pipeline, showing how a holistic scene latent is decomposed into individual object latents using a query-based transformer.

The core components include:

  • Holistic Scene Latent: The model starts with a compact, structured 3D latent ($z^s$) representing the entire scene, generated from an image or an existing mesh using a foundation model like TRELLIS.2.
  • Decomposition Transformer: This is a lightweight, query-based transformer. It takes the scene latent and a set of "queries" (which can be 2D masks, 3D points, or purely learnable slots) and learns to partition the scene latent into distinct object latents.
  • Feature Enhancement: To help the model distinguish between objects, the geometric latent is augmented with 3D semantic features from Utonia and 2D visual cues from DINOv3.
  • Parallel Decoding: Once the object latents are extracted, they are passed through a shared decoder in parallel. Because they were all carved out of the same global coordinate frame, the resulting meshes are naturally aligned without needing any post-hoc registration or pose estimation.

Datasets

The authors evaluated CoLa3D on a mix of synthetic and real-world benchmarks:

  • Synthetic: 3D-FRONT (professionally designed furnished rooms) and MetaScenes (simulatable scenes derived from real-world scans).
  • Real-World: Replica and ScanNet++, used for zero-shot generalization testing on challenging, scan-derived data.
  • Scalability: The model was also trained on ARSG-110K, a large-scale dataset with over 110k scenes, to test its performance on denser environments with up to 20 objects.

Key Insights and Ideas

  • The Power of Shared Coordinates: By performing decomposition in a shared latent space, spatial consistency is "baked in." The model naturally understands how objects relate to each other, drastically reducing physical interpenetration (clipping).
  • Extreme Data Efficiency: Remarkably, CoLa3D outperforms SAM 3D while using less than 1% of its training data. This suggests that the latent decomposition architecture provides a much stronger inductive bias for scene understanding than simply scaling up per-object generation.
  • Flexibility via Prompting: The query-based design allows the model to be "prompted" in various ways. You can give it a 2D mask from an image, click a 3D point in a scene, or let it automatically discover objects using learnable queries.

Key Results

CoLa3D achieves state-of-the-art performance across all tested benchmarks:

  • Speed: It is roughly $2\times$ faster than existing feed-forward baselines like MIDI or SceneGen because it avoids redundant per-object processing.
  • Accuracy: On the 3D-FRONT dataset, it reduced the scene-level Chamfer Distance (a measure of geometric error) by 27.9% compared to the best baseline. On real-world Replica data, the improvement was even more dramatic, with a 53.6% reduction in error.
  • Physical Plausibility: Qualitative results show that CoLa3D handles heavy occlusion and close object contact much better than baselines, which often produce distorted shapes when objects overlap.
Figure 1: Composable 3D scene generated by CoLa3D. Given a single input image (left), CoLa3D decomposes its whole-scene latent into individual object meshes (bottom) within a shared coordinate frame, faithfully reproducing the input layout (middle left) while keeping each object physically accessible for downstream tasks such as rearrangement (middle right) and physical simulation (right).

Limitations and Future Work

While a significant step forward, the authors note a few areas for improvement:

  • Latent Resolution: The current compact latent grid can be too coarse to capture very thin objects (like lamp stands) or distinguish between objects that are packed extremely tightly together. A hierarchical latent structure is suggested as a potential fix.
  • Data Bottleneck: Despite its efficiency, the model is still limited by the diversity of available 3D scene datasets, which are currently dominated by standard indoor furniture. Expanding to more diverse, "messy," or outdoor environments is a key next step.
  • Safety and Ethics: As with any high-fidelity reconstruction tool, there are concerns regarding the potential for creating "deepfake" environments or reconstructing private spaces from casual photos without consent.
Example of a composable 3D scene generated by CoLa3D, demonstrating its utility for downstream tasks like rearrangement and simulation.

Summary updated

Abstract

We introduce CoLa3D, a simple and effective model for composable 3D scene decomposition that, given a single whole-scene latent from either a single image or an existing scene mesh as input, directly outputs a set of object meshes with consistent spatial layout that can be assembled back into a coherent scene. CoLa3D introduces a novel paradigm for 3D scene decomposition within a shared whole-scene latent, unlike alternatives that perform per-object generation in canonical spaces and then glue the objects back together with fragile pose and scale estimation. The network is a lightweight, flexible, and scalable query-based transformer that accepts various types of queries, including 2D masks, 3D points, and learnable queries, and learns to decompose the scene into objects in a compact latent space. On both synthetic and real-world datasets, CoLa3D achieves state-of-the-art performance on composable 3D scene decomposition, substantially improving global scene-level consistency and object-level quality while running 2 times faster and using less than 1% of SAM 3D's training data. Visible depth on par with MoGe2 further confirms that latent decomposition preserves accurate visible scene geometry while producing physically complete, decomposable object meshes.
BibTeX
@misc{cai2026cola,
  author = {Xudong Cai and Yongcai Wang and Deying Li and Chuanxia Zheng},
  title = {{CoLa3D: Composable Latent 3D Decomposition}},
  year = {2026},
  eprint = {2611.03533},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03533},
}
Figure