Tetrahedral Diffusion Models for 3D Shape Generation

Nikolai Kalischek, Torben Peters, Jan D. Wegner, Konrad Schindler

TL;DR

This paper introduces TetraDiffusion, a novel diffusion model for generating high-resolution 3D shapes. By operating directly on a tetrahedral grid representation, it achieves significantly faster generation times and better geometric detail compared to existing methods, while also enabling textured 3D asset creation.

What Problem Does This Paper Address?

The generation of high-quality 3D assets is crucial for various applications, but it's a computationally intensive and often laborious process. While 2D diffusion models have seen great success, extending them to 3D has been challenging due to the inherent complexities of 3D representations. Existing 3D diffusion approaches often rely on voxel grids (memory-intensive and resolution-limited) or point clouds (lacking topological information). This paper addresses the need for a 3D representation that is both efficient and capable of capturing fine geometric details and topology, enabling high-resolution 3D shape generation with diffusion models.

Methods

The core innovation of TetraDiffusion lies in its use of a tetrahedral partitioning of 3D space as the primary representation. This structured, space-filling representation allows for:

  • Tetrahedral Convolutions: Novel convolution and transpose convolution operators designed to work directly on the tetrahedral grid. These operators are crucial for processing the geometric information within this structure.
  • Differentiable Marching Tetrahedra: This algorithm is used to extract surface meshes from the tetrahedral representation, and it's extended to handle additional attributes like color.
  • Hybrid Representation: Combining the advantages of point clouds (vertex flexibility) and structured grids (connectivity information), the model learns signed distance fields and per-vertex displacements on the tetrahedral grid.
  • Efficient Training and Inference: The tetrahedral representation allows for targeted grid pruning, significantly reducing memory consumption and computational cost, enabling high-resolution generation on consumer hardware.
  • Color Integration: The framework is extended to handle color attributes, allowing for the generation of textured 3D assets.

Datasets

The authors train and evaluate their model on the ShapeNet dataset, specifically using the airplane, bike, car, and chair classes.

Key Insights and Ideas

  • Tetrahedral Grids are a Sweet Spot: The paper demonstrates that a tetrahedral grid offers a compelling balance between the flexibility of point clouds and the structure of voxels, leading to efficient and high-quality 3D generation.
  • Direct Convolution on Tetrahedra is Key: Developing custom convolution operators that operate directly on the tetrahedral structure, rather than embedding it into a voxel grid, is critical for achieving efficiency and avoiding the limitations of voxel-based methods.
  • High Resolution is Achievable and Matters: TetraDiffusion's efficiency allows it to generate shapes at unprecedented resolutions, revealing finer geometric details that were previously a bottleneck for other methods.
  • Geometry and Texture Can Be Diffused Together: By extending the marching tetrahedra algorithm to handle vertex features beyond SDF and displacement, the model can jointly generate geometry and color, leading to consistent and detailed textured assets.

Key Results

  • Speed: TetraDiffusion is up to 200x faster than existing mesh diffusion techniques like MeshDiffusion. Inference takes approximately 3 seconds per sample.
  • Quality: The model generates high-resolution, detailed 3D shapes with sharper creases and fewer defects compared to other methods. Qualitative results showcase intricate details like chains and brake discs.
  • Resolution: TetraDiffusion can operate at substantially higher resolutions than existing mesh generators, achieving unprecedented spatial detail.
  • Efficiency: It significantly reduces memory consumption compared to voxel-based methods, allowing training on a single high-end GPU even at high resolutions.
  • Metrics: The model achieves competitive or superior performance across various metrics like 1-NNA, MMD, and COV, and excels in Shading-FID scores.

Limitations and Future Work

  • Limitations:
    • The paper notes that while the model can be guided by color names, the results might slightly degrade due to the discrepancy between training and inference signals, especially with noisy meshes.
  • The comparison with point cloud generators is acknowledged as non-trivial due to differences in point cloud densities.
  • Future Work:
    • Leveraging the extended differentiable marching tetrahedra algorithm and differentiable rendering for seamless integration of large-scale 2D data.
  • Extending the framework to synthesize entire scenes, which often exhibit larger extents and higher sparsity.

Summary updated

Abstract

Recently, probabilistic denoising diffusion models (DDMs) have greatly advanced the generative power of neural networks. DDMs, inspired by non-equilibrium thermodynamics, have not only been used for 2D image generation, but can also readily be applied to 3D point clouds. However, representing 3D shapes as point clouds has a number of drawbacks, most obvious perhaps that they have no notion of topology or connectivity. Here, we explore an alternative route and introduce tetrahedral diffusion models, an extension of DDMs to tetrahedral partitions of 3D space. The much more structured 3D representation with space-filling tetrahedra makes it possible to guide and regularize the diffusion process and to apply it to colorized assets. To manipulate the proposed representation, we develop tetrahedral convolutions, down- and up-sampling kernels. With those operators, 3D shape generation amounts to learning displacement vectors and signed distance values on the tetrahedral grid. Our experiments confirm that Tetrahedral Diffusion yields plausible, visually pleasing and diverse 3D shapes, is able to handle surface attributes like color, and can be guided at test time to manipulate the resulting shapes.
BibTeX
@inproceedings{kalischek2024tetrahedral,
  author = {Nikolai Kalischek and Torben Peters and Jan D. Wegner and Konrad Schindler},
  title = {{Tetrahedral Diffusion Models for 3D Shape Generation}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73668-1\_21},
}
Figure