DDSL: Deep Differentiable Simplex Layer for Learning Geometric Signals

Chiyu Max Jiang, Dana Lynn Ona Lansigan, Philip Marcus, Matthias Nießner

Introduction

In geometric deep learning, a fundamental divide exists between meshes (flexible, non-uniform representations like point clouds or wireframes) and grids (structured data like 2D images or 3D voxels). While grids are the native language of Convolutional Neural Networks (CNNs), meshes are often the preferred format for physical modeling and computer graphics. Bridging these two worlds requires rasterization—the process of converting a mesh into a grid.

However, standard rasterization is typically a discrete, non-differentiable operation. Small shifts in a vertex's position can cause abrupt changes in pixel values, creating discontinuities that break the gradient flow necessary for backpropagation. The Deep Differentiable Simplex Layer (DDSL) introduces a mathematically rigorous framework to perform "in-situ" differentiable rasterization, allowing for end-to-end gradient-based optimization of geometric shapes across various dimensions.

The DDSL Framework

The core innovation of DDSL is the Simplex Non-Uniform Fourier Transform (NUFT). Rather than relying on traditional rendering pipelines like Z-buffering, DDSL treats a mesh as a continuous signal and operates in the frequency domain.

Forward and Backward Passes

  1. Forward Pass: The layer accepts a simplex mesh (defined by its vertices, connectivity, and density) and computes its Fourier coefficients. To prevent aliasing and "ringing" artifacts, a Gaussian filter is applied. Finally, an Inverse Fast Fourier Transform (iFFT) generates the physical raster image.
  2. Backward Pass: The authors derived the analytical derivative of the simplex NUFT. This allows the network to calculate precisely how a change in a pixel's intensity should translate to a movement of a specific vertex. This analytical approach provides a 10x to 100x speedup over standard numerical finite-difference methods.

Simplex-Degree Agnosticism

Unlike many differentiable renderers that are specialized for 3D-to-2D projection of triangles, DDSL is "simplex-degree agnostic." It utilizes the same theoretical framework to handle points ($j=0$), lines ($j=1$), triangles ($j=2$), and tetrahedra ($j=3$). This allows for rasterization in the same dimension as the input, which is vital for tasks like volumetric analysis or 2D shape optimization.

Applications and Results

The versatility of DDSL was validated across several distinct domains, ranging from computer vision to aerodynamic engineering.

Polygonal Image Segmentation

On the Cityscapes dataset, the authors introduced "PolygonNet," a simplified architecture enabled by DDSL. Because the layer is fully differentiable, PolygonNet avoids the complex Reinforcement Learning or Graph Neural Network pipelines used by previous models like Polygon-RNN++.

  • Performance: PolygonNet achieved a state-of-the-art 72.50 Mean IoU (Intersection over Union).
  • Efficiency: The model is 100x faster at runtime and requires only 25% of the parameters compared to previous benchmarks.

Aerodynamic Shape Optimization

Using data from Airfoiltools.com, DDSL was employed to optimize aircraft wing profiles. The layer allowed a neural network to "evolve" a standard airfoil into a specific target shape to achieve desired lift-drag ratios, demonstrating its utility in physics-based design tasks where precise geometric control is mandatory.

3D Mesh Estimation and Regularization

In 3D tasks using ShapeNet, DDSL helped solve common issues in mesh estimation. While standard losses like Chamfer distance often struggle with mesh topology, DDSL provides a loss based on the enclosed volume. This ensures that generated meshes are "watertight" and correctly oriented.

Discussion and Limitations

While DDSL represents a significant leap in differentiable geometry, it is not without challenges. The authors note a sensitivity to initialization; because the loss landscape is not globally convex, optimization can occasionally stall in local minima if the initial mesh guess is significantly far from the target. Additionally, while the NUFT is efficient, its reliance on spectral computations may face scaling hurdles when applied to extremely high-resolution 3D grids.

Conclusion

DDSL provides a robust mathematical bridge between the flexibility of meshes and the power of grid-based deep learning. By enabling analytical gradients for any simplex type, it simplifies the training of geometric models and opens new doors for high-performance segmentation and automated physical design.

Summary updated

Abstract

We present a Deep Differentiable Simplex Layer (DDSL) for neural networks for geometric deep learning. The DDSL is a differentiable layer compatible with deep neural networks for bridging simplex mesh-based geometry representations (point clouds, line mesh, triangular mesh, tetrahedral mesh) with raster images (e.g., 2D/3D grids). The DDSL uses Non-Uniform Fourier Transform (NUFT) to perform differentiable, efficient, anti-aliased rasterization of simplex-based signals. We present a complete theoretical framework for the process as well as an efficient backpropagation algorithm. Compared to previous differentiable renderers and rasterizers, the DDSL generalizes to arbitrary simplex degrees and dimensions. In particular, we explore its applications to 2D shapes and illustrate two applications of this method: (1) mesh editing and optimization guided by neural network outputs, and (2) using DDSL for a differentiable rasterization loss to facilitate end-to-end training of polygon generators. We are able to validate the effectiveness of gradient-based shape optimization with the example of airfoil optimization, and using the differentiable rasterization loss to facilitate end-to-end training, we surpass state of the art for polygonal image segmentation given ground-truth bounding boxes.
BibTeX
@inproceedings{jiang2019ddsl,
  author = {Chiyu Max Jiang and Dana Lynn Ona Lansigan and Philip Marcus and Matthias Nießner},
  title = {{DDSL: Deep Differentiable Simplex Layer for Learning Geometric Signals}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00886},
}
Figure