Appearance-Driven Automatic 3D Model Simplification

Jon Hasselgren, Jacob Munkberg, Jaakko Lehtinen, Miika Aittala, Samuli Laine

Abstract

We present a suite of techniques for jointly optimizing triangle meshes and shading models to match the appearance of reference scenes. This capability has a number of uses, including appearance-preserving simplification of extremely complex assets, conversion between rendering systems, and even conversion between geometric scene representations. We follow and extend the classic analysis-by-synthesis family of techniques: enabled by a highly efficient differentiable renderer and modern nonlinear optimization algorithms, our results are driven to minimize the image-space difference to the target scene when rendered in similar viewing and lighting conditions. As the only signals driving the optimization are differences in rendered images, the approach is highly general and versatile: it easily supports many different forward rendering models such as normal mapping, spatially-varying BRDFs, displacement mapping, etc. Supervision through images only is also key to the ability to easily convert between rendering systems and scene representations. We output triangle meshes with textured materials to ensure that the models render efficiently on modern graphics hardware and benefit from, e.g., hardware-accelerated rasterization, ray tracing, and filtered texture lookups. Our system is integrated in a small Python code base, and can be applied at high resolutions and on large models.

Introduction

In modern computer graphics, 3D assets are becoming incredibly complex, often consisting of millions or even billions of triangles. To render these efficiently in real-time applications like games or interactive simulations, developers create "Levels of Detail" (LODs)—simpler versions of the models used when the object is far from the camera. Traditionally, simplification has been a purely geometric process, collapsing edges to minimize the distance between the old and new surfaces. However, small geometric errors can lead to massive visual artifacts, such as "popping" textures or broken silhouettes.

This research argues that because the ultimate goal of 3D rendering is a 2D image on a screen, we should use image-space error to drive the simplification of both shape and appearance simultaneously. By optimizing for how a model looks rather than just its geometric shape, we can achieve far more efficient representations.

Methodology: Analysis-by-Synthesis

The core of the framework is an analysis-by-synthesis approach powered by a high-performance differentiable rendering pipeline. The process begins with a "latent representation" consisting of a low-resolution mesh and a set of material textures (diffuse, specular, normals, and displacement maps).

The analysis-by-synthesis pipeline: A latent representation is rendered through a differentiable pipeline and compared against a reference to minimize image-space loss.

Unlike standard renderers, a differentiable one allows the system to calculate exactly how changing a single vertex position or a single pixel in a texture map will affect the final rendered image. The optimization loop proceeds as follows:

  1. Rendering: The system renders the simplified model and a high-quality reference model from random viewpoints and lighting conditions.
  2. Loss Calculation: It calculates the visual similarity (loss) between the two images.
  3. Optimization: Using stochastic gradient descent (specifically the Adam optimizer), the system "nudges" the mesh vertices and texture pixels to minimize the loss.
  4. Joint Optimization: The system automatically "negotiates" between different representations. If a detail is too fine for the low-poly mesh to capture, the optimizer naturally bakes that detail into the normal or displacement maps.
  5. Regularization: To maintain surface integrity, a Laplacian regularizer is used to prevent the mesh from becoming overly distorted during the optimization process.

Versatility Across Representations

Because the system only requires images of the reference, the reference itself acts as a "black box." It does not need to be a mesh; it can be a path-traced scene, a volumetric cloud, or even an implicit mathematical formula like a Signed Distance Field (SDF). This makes the tool a powerful "universal translator" for 3D representations, capable of converting complex ray-marched ShaderToy objects into standard textured meshes compatible with traditional game engines.

Key Results

The method demonstrates significant improvements over industry-standard tools like Simplygon, achieving higher Peak Signal-to-Noise Ratio (PSNR) and better visual fidelity for the same triangle budgets.

Extreme simplification of aggregate geometry: Our method reduces a 5.1 billion triangle reference to 20 million triangles (0.4%) while maintaining visual fidelity.

One of the most striking results is the simplification of aggregate geometry, such as the foliage in Disney’s Moana Island scene. The system reduced the triangle count to just 0.4% of the reference (from 5.1 billion to 20 million triangles) while maintaining a nearly identical look. Additionally, the method successfully optimized skinning weights for animated characters, ensuring that simplified models remain visually consistent even during complex movement.

Limitations and Future Work

While highly effective, the system currently requires some manual tuning of hyperparameters like learning rates and regularization weights. There are also inherent limits to representing 3D depth with 2D textures; for example, view parallax can cause textures to appear blurry from extreme angles. Furthermore, the current implementation can move vertices but cannot change the topology of the mesh (e.g., creating new holes). Future work may involve integrating topology-changing operations and extending the internal renderer to handle more complex global illumination effects during the optimization loop.

Summary updated

Abstract

We present a suite of techniques for jointly optimizing triangle meshes and shading models to match the appearance of reference scenes. This capability has a number of uses, including appearance-preserving simplification of extremely complex assets, conversion between rendering systems, and even conversion between geometric scene representations. We follow and extend the classic analysis-by-synthesis family of techniques: enabled by a highly efficient differentiable renderer and modern nonlinear optimization algorithms, our results are driven to minimize the image-space difference to the target scene when rendered in similar viewing and lighting conditions. As the only signals driving the optimization are differences in rendered images, the approach is highly general and versatile: it easily supports many different forward rendering models such as normal mapping, spatially-varying BRDFs, displacement mapping, etc. Supervision through images only is also key to the ability to easily convert between rendering systems and scene representations. We output triangle meshes with textured materials to ensure that the models render efficiently on modern graphics hardware and benefit from, e.g., hardware-accelerated rasterization, ray tracing, and filtered texture lookups. Our system is integrated in a small Python code base, and can be applied at high resolutions and on large models. We describe several use cases, including mesh decimation, level of detail generation, seamless mesh filtering and approximations of aggregate geometry.
BibTeX
@inproceedings{hasselgren2021appearance,
  author = {Jon Hasselgren and Jacob Munkberg and Jaakko Lehtinen and Miika Aittala and Samuli Laine},
  title = {{Appearance-Driven Automatic 3D Model Simplification}},
  booktitle = {EGSR (DL)},
  year = {2021},
  doi = {10.2312/sr.20211293},
}
Figure