Appearance-Driven Automatic 3D Model Simplification

Jon Hasselgren, Jacob Munkberg, Jaakko Lehtinen, Miika Aittala, Samuli Laine

TL;DR

NVIDIA researchers have developed a framework that automatically simplifies massive 3D models by optimizing them to look like the original, rather than just approximating their geometric shape. By using a differentiable renderer, the system jointly optimizes mesh geometry and material textures (like normal and displacement maps) to minimize visual differences, achieving extreme compression—such as representing billions of triangles with just 0.4% of the original count—without losing visual fidelity.

What Problem Does This Paper Address?

In modern computer graphics, 3D assets are becoming incredibly complex, often consisting of millions or even billions of triangles. To render these efficiently in real-time (like in games), developers create "Levels of Detail" (LODs)—simpler versions of the models used when the object is far away.

Traditionally, simplification is a geometric process: you collapse edges to minimize the distance between the old and new surfaces. However, small geometric errors can lead to massive visual artifacts, like "popping" textures or broken silhouettes. Furthermore, the mesh and the materials (textures) are usually handled separately. This paper argues that because the ultimate goal is a 2D image on a screen, we should use image-space error to drive the simplification of both shape and appearance simultaneously.

Methods

The authors employ an analysis-by-synthesis approach powered by differentiable rendering.

  1. Latent Representation: The model starts with a "latent" version of the asset—a low-resolution mesh and a set of textures (diffuse, specular, normals, etc.).
  2. Differentiable Pipeline: They use a custom, high-performance differentiable rasterizer. Unlike standard renderers, a differentiable one allows the system to calculate exactly how changing a single vertex position or a single pixel in a texture map will affect the final rendered image.
  3. Optimization Loop: The system renders the simplified model and a high-quality reference model from random viewpoints and lighting conditions. It calculates the difference (loss) between the two images and uses stochastic gradient descent (specifically the Adam optimizer) to "nudge" the mesh vertices and texture pixels until the images match.
  4. Joint Optimization: Crucially, the system "negotiates" between different representations. If a detail is too small for the low-poly mesh to handle, the optimizer naturally "bakes" that detail into the normal map or displacement map instead.
  5. Regularization: To prevent the mesh from becoming a "spiky" mess during optimization, they use a Laplacian regularizer to maintain surface smoothness.

Datasets

The paper demonstrates the versatility of the method across several high-quality datasets:

  • Disney’s Moana Island Scene: Used for "aggregate geometry" like complex foliage and trees.
  • Smithsonian 3D Digitization Project: High-resolution scans of sculptures (the "Ewer" and "Dancer") used for testing normal map baking and LOD generation.
  • Unreal Engine Paragon Assets: Used to demonstrate "automatic cleanup" of poorly decimated character meshes.
  • RenderPeople: Rigged and animated characters used to show that the method works for skinning and animation weights.
  • ShaderToy (Implicit Surfaces): Used to demonstrate converting ray-marched Signed Distance Fields (SDFs) into standard textured meshes.

Key Insights and Ideas

  • Appearance over Geometry: The core "aha moment" is that the geometric "correctness" of a simplified mesh matters less than its visual impact. By focusing on the rendered result, the system can make intelligent trade-offs that traditional tools miss.
  • The "Black Box" Reference: Because the system only needs images of the reference, the reference doesn't have to be a mesh. It can be a path-traced scene, a volumetric cloud, or an implicit mathematical formula. This makes the tool a powerful "universal translator" for 3D representations.
  • Automatic Division of Labor: The joint optimization of displacement maps, normal maps, and vertex positions allows the system to automatically decide which frequency of detail belongs in which map. This replaces hours of manual artist labor in "baking" textures.

Key Results

  • Extreme Simplification: For the Moana foliage, the system reduced the triangle count to just 0.4% of the reference while maintaining a nearly identical look when rendered in a path tracer.
  • Outperforming Industry Standards: In head-to-head comparisons with Simplygon (the industry standard for LOD generation), the proposed method achieved higher PSNR (Peak Signal-to-Noise Ratio) and significantly better visual quality for the same triangle budget.
  • Animation Fidelity: The method successfully optimized skinning weights for animated characters, ensuring that simplified models don't "break" or look distorted when they move.
  • SDF Conversion: It successfully converted complex ray-marched "ShaderToy" objects into standard meshes that can be used in any traditional game engine.

Limitations and Future Work

  • Hyperparameter Tuning: The system currently requires some manual tuning of learning rates and regularization weights depending on the specific asset.
  • View Parallax: When simplifying 3D structures into flat quads (like leaves), the textures can appear blurry from certain angles because a 2D texture cannot perfectly represent 3D depth (parallax).
  • Topology Constraints: The method can move vertices but cannot currently change the "topology" of the mesh (e.g., it cannot turn a sphere into a donut by poking a hole in it).
  • Rasterization Limits: Since the internal renderer is rasterization-based, it doesn't natively handle complex global illumination or refractions during the optimization loop, though it can "bake" these effects if they are present in the reference images.

Summary updated

Abstract

We present a suite of techniques for jointly optimizing triangle meshes and shading models to match the appearance of reference scenes. This capability has a number of uses, including appearance-preserving simplification of extremely complex assets, conversion between rendering systems, and even conversion between geometric scene representations. We follow and extend the classic analysis-by-synthesis family of techniques: enabled by a highly efficient differentiable renderer and modern nonlinear optimization algorithms, our results are driven to minimize the image-space difference to the target scene when rendered in similar viewing and lighting conditions. As the only signals driving the optimization are differences in rendered images, the approach is highly general and versatile: it easily supports many different forward rendering models such as normal mapping, spatially-varying BRDFs, displacement mapping, etc. Supervision through images only is also key to the ability to easily convert between rendering systems and scene representations. We output triangle meshes with textured materials to ensure that the models render efficiently on modern graphics hardware and benefit from, e.g., hardware-accelerated rasterization, ray tracing, and filtered texture lookups. Our system is integrated in a small Python code base, and can be applied at high resolutions and on large models. We describe several use cases, including mesh decimation, level of detail generation, seamless mesh filtering and approximations of aggregate geometry.
BibTeX
@inproceedings{hasselgren2021appearance,
  author = {Jon Hasselgren and Jacob Munkberg and Jaakko Lehtinen and Miika Aittala and Samuli Laine},
  title = {{Appearance-Driven Automatic 3D Model Simplification}},
  booktitle = {EGSR (DL)},
  year = {2021},
  doi = {10.2312/sr.20211293},
}

Loading related content...

Figure