SpaceControl: Introducing Test-Time Spatial Control to 3D Generative Modeling

Elisabetta Fedele, Francis Engelmann, Ian Huang, Or Litany, Marc Pollefeys, Leonidas Guibas

TL;DR

SpaceControl is a training-free, test-time method that grants users precise spatial control over 3D generative models using geometric "sketches." By intervening in the latent space of pre-trained models like Trellis, it allows for fine-grained editing and generation of high-quality 3D assets from simple primitives or meshes—all without the need for costly retraining or slow optimization.

SpaceControl enables both spatial controlled generation from coarse primitives and spatial controlled editing of existing assets.

The Challenge of 3D Controllability

While 3D generative AI has made massive leaps in quality, controllability remains a major hurdle. Most current models rely heavily on text or image prompts. Text is notoriously ambiguous—try describing the exact curvature of a chair's leg in words—and 2D images are difficult to manipulate if you want to change a specific 3D dimension.

Existing solutions for spatial control usually fall into two camps, both with significant drawbacks:

  1. Training-based: These require fine-tuning a model on specific geometric inputs (like voxels or bounding boxes). This is computationally expensive and often degrades the model's ability to generalize to new, unseen shapes.
  2. Optimization-based: These guide the generation process at inference time but are often painfully slow, requiring many iterations to align the output with the control shape.

SpaceControl aims to provide the "best of both worlds": the speed and quality of a pre-trained model combined with the precise, "hands-on" control of a traditional 3D modeling tool.

How SpaceControl Works

SpaceControl is built on the foundation of Rectified Flow models, specifically leveraging the state-of-the-art Trellis architecture. The core innovation is a latent-space intervention technique that guides generation without altering the model's weights.

  • The Input: Users provide a 3D geometric "guide." This could be a collection of Superquadrics (mathematically flexible primitives like rounded cubes or ellipsoids) or a full polygon mesh.
  • Voxelization & Encoding: This guide is voxelized and passed through the model's pre-trained encoder to map it into the latent space.
  • The "Noising" Trick: Instead of starting the generation process from pure random noise, the method takes the encoded guide and "noises it up" to a specific time step ($t_0$).
  • Denoising with Guidance: The model then denoises this signal back into a clean 3D asset. Because the process starts from a "noisy version" of the user's shape, the final output naturally adheres to that geometry while the generative model fills in the realistic details, textures, and fine structures.
  • The Realism-Faithfulness Knob: A hyperparameter ($\tau_0$) allows users to dial the strength of the control. A high value forces the AI to stick strictly to the user's sketch; a lower value gives the AI more "creative freedom" to produce a more realistic, albeit less strictly faithful, object.

Key Insights and Innovations

  • Training is Optional: The most significant takeaway is that we don't need to retrain massive 3D models to give them spatial awareness. By understanding the latent trajectory of Rectified Flow models, we can simply "nudge" the generation process at the right moment.
  • Superquadrics as the "UI" for 3D AI: The paper highlights superquadrics as an ideal interface. They are more expressive than simple bounding boxes but much easier for a human to manipulate than a raw point cloud or dense mesh.
  • Multi-Modal Synergy: SpaceControl doesn't just use geometry; it integrates seamlessly with text and image prompts. The geometry defines the where and what shape, while text or images define the style and texture.

Performance and Results

The researchers evaluated SpaceControl using standard benchmarks like ShapeNet (Chairs/Tables) and Toys4K, comparing it against both training-based (Spice-E) and optimization-based (Coin3D) baselines.

  • Geometric Faithfulness: Measured by Chamfer Distance (how closely the output matches the input shape), SpaceControl outperformed baselines significantly. On the Toys4K dataset, it achieved a score of 14.0, compared to 54.4 for Coin3D and 65.9 for Spice-E.
  • Visual Quality: Despite strict geometric constraints, the model maintained high realism. Its FID scores (measuring image quality) were comparable to or better than unconstrained models.
  • User Preference: In a study with 52 volunteers, SpaceControl was preferred 85% of the time for overall appearance and alignment compared to the training-based Spice-E baseline.

Limitations and Future Directions

While highly effective, SpaceControl has a few areas ripe for future research:

  • Manual Tuning: Currently, the user must manually adjust the $\tau_0$ parameter to find the sweet spot between realism and faithfulness. Future work could involve "adaptive scheduling" to automate this balance.
  • Global vs. Local Control: The current implementation applies control strength globally. "Part-aware" control—where a user could be very strict about a chair's seat but let the AI be creative with the legs—is a promising next step.
  • Scene-Level Generation: While this paper focuses on individual assets, early tests of scene-level generation (e.g., a whole room layout) suggest this technique could eventually power entire virtual environment design workflows.

Summary updated

Abstract

Generative methods for 3D assets have recently achieved remarkable progress, yet providing intuitive and precise control over the object geometry remains a key challenge.Existing approaches predominantly rely on text or image prompts, which often fall short in geometric specificity: language can be ambiguous, and images are difficult to manipulate. In this work, we introduce SpaceControl, a training-free test-time method for explicit spatial control of 3D asset generation. Our approach accepts a wide range of geometric inputs, from coarse primitives to detailed meshes, and integrates seamlessly with modern generative models without requiring any additional training. A control parameter lets users trade off between geometric fidelity and output realism. Extensive quantitative evaluation and user studies demonstrate that SpaceControl outperforms both training-based and optimization-based baselines in geometric faithfulness while preserving high visual quality. Finally, we present an interactive interface for real-time superquadric editing and direct 3D asset generation, enabling seamless use in creative workflows. Project page: https://spacecontrol3d.github.io/
BibTeX
@inproceedings{fedele2026spacecontrol,
  author = {Elisabetta Fedele and Francis Engelmann and Ian Huang and Or Litany and Marc Pollefeys and Leonidas Guibas},
  title = {{SpaceControl: Introducing Test-Time Spatial Control to 3D Generative Modeling}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=mEqsCVI5sN},
}
Figure