Introduction
Generative AI has revolutionized 3D content creation, yet a significant gap remains between coarse conceptual models and the intricate, high-frequency geometric details required for professional applications. While a model might capture the general silhouette of a chair, it often lacks the fine-grained textures—such as ornate moldings or complex latticework—that define high-quality assets. DECOLLAGE (3D Detailization by Controllable, Localized, and Learned Geometry Enhancement) addresses this by providing a framework to "paint" geometric details from exemplar shapes onto coarse voxel inputs.
The Problem: Global Constraints and Structural Fragility
Existing detailization methods typically struggle with two primary issues:
- Lack of Localized Control: Most systems apply a single style globally. Users cannot easily combine disparate styles, such as applying ornate baroque legs to a minimalist modern seat.
- Structural Instability: When mixing styles or applying them to "out-of-distribution" shapes (shapes the model hasn't seen), the geometry often "shatters." This results in disconnected components or floating artifacts that violate the physical integrity of the object.
DECOLLAGE is designed to be both interactive and robust, ensuring that even when styles are mixed, the resulting object remains structurally sound and faithful to the original design.
Methodology: The Pyramid GAN Architecture
The technical foundation of DECOLLAGE is a Pyramid GAN that progressively upsamples coarse voxel inputs through multiple levels, from a resolution of $16^3$ to $256^3$. This hierarchical approach allows the model to separate global structure from local texture.
Masking-Aware Generation and Style Mixing
The framework utilizes part labels and style codes to enable localized enhancement. By assigning different style codes to specific regions (e.g., the backrest vs. the legs), users can achieve complex style mixing. This allows for a "painting" workflow where users can selectively apply geometric textures to different parts of a model.
Adaptive α Weighting
To solve the "shattering" problem at part boundaries, the authors introduced Adaptive α Weighting. Within the center of a part, the network prioritizes a style discriminator to maximize geometric detail. However, near the transition boundaries where two parts meet, the network automatically shifts priority toward a global discriminator. This mechanism ensures that different geometric textures connect realistically, maintaining topological continuity and preventing floating pieces.
Structure-Preserving Losses
To ensure the high-resolution output does not deviate from the user's intended design, DECOLLAGE employs two specific losses:
- $\mathcal{L}_{down}$: Forces the downsampled version of the high-res output to match the coarse input.
- $\mathcal{L}_{up}$: Ensures consistency during the upsampling process.
These losses prevent the AI from "hallucinating" entirely new shapes, keeping the detailization grounded in the original voxel structure.
Experimental Results
The researchers evaluated DECOLLAGE across diverse categories from ShapeNet (chairs, tables, plants) and 3D Warehouse (buildings, cakes, crystals). Each shape was manually segmented into functional parts to facilitate localized training.
Quantitative Performance
DECOLLAGE significantly outperforms previous benchmarks like DECOR-GAN and ShaDDR. In terms of "Strict-IOU"—a metric measuring how well the output adheres to the input structure—DECOLLAGE achieved a score of 0.748, a substantial improvement over DECOR-GAN’s 0.581.
Qualitative Connectivity and Speed
Qualitatively, the model excels at maintaining connectivity. Unlike previous methods that often leave "floating" pieces when styles conflict, DECOLLAGE successfully bridges the gaps at joints. Furthermore, the model is highly efficient; once trained, it can generate a detailed 3D shape in approximately 0.3 seconds, making it fast enough for real-time interactive applications.
Limitations and Future Directions
Despite its strengths, DECOLLAGE has limitations. It currently relies on pre-defined semantic part labels, and mixing highly dissimilar categories (e.g., plant textures on a chair) can still produce artifacts. Additionally, while $256^3$ is high for voxel representations, it may not suffice for micro-textures or jewelry-level detail.
Future work points toward integrating DECOLLAGE with Large Language Models (LLMs) to allow for text-prompted detailization (e.g., "make the legs look Gothic") and exploring diffusion-based upsampling to push resolution boundaries even further.
Summary updated