MoCA: Mixture-of-Components Attention for Scalable Compositional 3D Generation

Zhiqi Li, Wenhuan Li, Tengfei Wang, Zhenwei Wang, Junta Wu, Haoyuan Wang, Yunhan Yang, Zehuan Huang, Yang Li, Peidong Liu, Chunchao GuoSubmission to ICLR 2026 · Rejected

TL;DR

Generating complex 3D objects and scenes composed of many distinct parts has traditionally hit a severe computational wall due to the quadratic cost of global attention. MoCA (Mixture-of-Components Attention) breaks this barrier by routing full-token attention only to the top-$k$ most relevant components while compressing distant ones into compact tokens. This allows the model to scale cleanly up to 32 components—doubling the capacity of prior frameworks—while improving geometric fidelity and decomposition accuracy.

What Problem Does This Paper Address?

Monolithic 3D generative models produce single, indivisible meshes, making downstream tasks like part animation, component-level editing, or physics simulation extremely difficult. To enable compositionality, recent works attempt to generate multiple parts or scene instances concurrently using diffusion transformers (DiTs).

However, they rely on naive global attention across all components. For $N$ components each represented by $L$ latent tokens, global attention scales quadratically as $\mathcal{O}(N^2 L^2)$. Consequently, prior methods choke beyond a handful of components (often fewer than 8 to 20). Furthermore, uniform attention is wasteful: a character's hand only needs fine-grained geometric context from the wrist and forearm, not from distant boots. Blindly attending across all tokens wastes capacity on low-value interactions.

Methods

MoCA structures the denoising process by interleaving local component-level attention with a new Mixture-of-Components (MoC) global attention block:

[Component Tokens] 
       │
       ├──> Anchor Token (Aggregated Representation) ──> Router (Sigmoid)
       │                                                    │
       ├──> Top-k Important Components ──────────────────> Full Tokens (zj)
       │                                                    │
       └──> Unimportant Components ──────────────────────> Compressed Tokens (pj)
                                                            │
                                             MoC Attention (Reweighted Keys)
  • Learnable Queries & Token Compression: To compress unstructured 3D vecset latents, MoCA appends $N_p = L / \sigma$ learnable query tokens (compression ratio $\sigma=8$) and an anchor token $\bar{p}$ per component via cross-attention.
  • Component-Level Router: Instead of Softmax, the router projects anchor tokens into queries and keys and applies a Sigmoid activation to score pairwise component relevance, avoiding vanishing logits in large scenes.
  • Selective MoC Attention: For any given component, it attends to the full tokens $z_j$ of the top-$k$ most relevant components, and only the compressed tokens $p_j$ for the remaining distant ones.
  • Key-Vector Gating: Predicted importance scores multiply directly into the Key vectors of routed components, allowing router gradients to train end-to-end.
  • Auxiliary-Loss-Free Load Balancing: To prevent premature collapse where the router over-relies on a few components early in training, MoCA stochastically samples components based on importance logits during training, reverting to deterministic top-$k$ at inference.
  • Multi-Head Routing: Routing decisions are computed independently per attention head to capture varied relational dependencies.

Datasets

  • Pretraining & Ablation: Pretrained on a curated object dataset of ~2 million watertight meshes with part annotations (supporting up to 32 parts per asset). Ablations were conducted on Trellis-500K.
  • Scene Adaptation: Fine-tuned on 3D-FRONT, augmented with four newly rendered isometric multi-view images with randomized floor and wall textures to handle complex indoor rooms.
  • Evaluation Benchmarks: Evaluated on PartObjaverse-Tiny (200 shapes) and ABO (100 objects) for part-level decomposition, and the 3D-FRONT test split for scene generation.

Key Insights and Ideas

  • Don't discard distant components—compress them: Ablation studies show that dropping unselected components entirely causes a massive drop in quality (Chamfer Distance worsens from 0.1180 to 0.1523). Retaining compressed tokens preserves coarse global layout and spatial context while saving massive amounts of compute.
  • Pre-softmax gating maintains stability: Multiplying router scores into Key vectors acts as pre-softmax reweighting. Gating Value vectors post-softmax caused performance to collapse because attention weights no longer summed to 1, introducing numerical instability.
  • Sigmoid beats Softmax for routing: Softmax forces competition across all $N$ components, leading to vanishingly small weights when $N$ is large. Independent Sigmoid gates ensure that multiple closely connected parts can all receive appropriately high attention weights.

Key Results

  • Part-Aware Object Generation: On PartObjaverse-Tiny, MoCA achieved a Chamfer Distance (CD) of 0.1010 and an [email protected] of 0.8708, outperforming PartPacker (0.1105 / 0.8484) and PartCrafter (0.1195 / 0.8169) with cleaner separations and fewer floating artifacts.
  • Complex Scene Generation: On 3D-FRONT, MoCA reached a scene CD of 0.1201 and F-score of 0.8637 (vs. MIDI's 0.1548 / 0.8238). It successfully synthesized cluttered scenes with >16 instances, where prior baselines are typically constrained to fewer than 8.
  • Computational Efficiency: At $N=32$ parts (1024 tokens each), MoCA cut global attention runtime nearly in half compared to PartCrafter (79 ms vs. 142 ms on an H20 GPU).

Limitations and Future Work

  • Frozen Latent Space: MoCA currently uses a frozen vecset VAE. When an asset contains dozens of components, individual components occupy very small global bounding volumes, hitting the resolution limit of the frozen autoencoder. Future work will explore component-level VAE fine-tuning.
  • Router Implementation Overhead: While attention computation scales efficiently, the current PyTorch implementation of the routing/indexing logic introduces substantial latency (36 ms at $N=32$), highlighting the need for custom CUDA kernels.

Summary updated

Abstract

Compositionality is critical for 3D object and scene generation, but existing part-aware 3D generation methods suffer from poor scalability due to quadratic global attention costs when increasing the number of components. In this work, we present MoCA, a compositional 3D generative model with two key designs: 1) importance-based component routing that selects top-k relevant components for sparse global attention, and 2) unimportant components compression that preserve contextual priors of unselected components while reducing computational complexity of global attention. With these designs, MoCA enables efficient, fine-grained compositional 3D asset creation with scalable number of components. Extensive experiments show MoCA outperforms baselines on both compositional object and scene generation tasks.
BibTeX
@misc{li2025moca,
  author = {Zhiqi Li and Wenhuan Li and Tengfei Wang and Zhenwei Wang and Junta Wu and Haoyuan Wang and Yunhan Yang and Zehuan Huang and YANG LI and Peidong Liu and Chunchao Guo},
  title = {{MoCA: Mixture-of-Components Attention for Scalable Compositional 3D Generation}},
  year = {2025},
  eprint = {2512.07628},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.07628},
}
Figure