NeuMIP: multi-resolution neural materials

Alexandr Kuznetsov, Krishna Mullia, Zexiang Xu, Milos Hasan, Ravi Ramamoorthi

NeuMIP introduces a novel neural framework designed to render complex materials—such as knitted wool, rocky surfaces, or insulation foam—across multiple scales with high fidelity. By integrating a "neural texture pyramid" with an unsupervised "neural offset" module, the system achieves realistic parallax and self-shadowing effects without requiring explicit geometry or heightfields. Notably, NeuMIP maintains this visual quality while utilizing significantly fewer parameters than previous state-of-the-art methods.

The Challenge of Multi-Scale Material Rendering

A classic challenge in computer graphics is rendering materials consistently at varying distances, known as level-of-detail (LOD) management. While standard "mipmapping" effectively handles simple color averaging for distant objects, it fails for complex materials where appearance is dictated by intricate 3D structures, or mesostructures.

For instance, when viewing a knitted sweater from afar, individual fibers blend into yarns, which eventually appear as a continuous fabric. Traditional methods struggle to "prefilter" these non-linear visual effects. While high-fidelity simulations like Monte Carlo path tracing can model every fiber, they are computationally expensive and produce significant noise at a distance. NeuMIP bridges this gap by providing a neural representation that is both multi-resolution and capable of efficiently handling complex 3D-like visual effects.

Methods: The MBTF Architecture

The core of the paper is the Multi-scale Bidirectional Texture Function (MBTF). This 7-dimensional function takes surface position, light direction, view direction, and a filter kernel size (representing the zoom level) to produce an RGB color.

The NeuMIP architecture consists of a Neural Offset Module, a Neural Texture Pyramid, and an MLP Decoder to generate final RGB values.

The architecture is composed of three primary components:

  1. Neural Offset Module: This module is responsible for simulating 3D effects. It predicts a coordinate offset based on the viewing angle, shifting the texture lookup to simulate depth and parallax. This module is trained in an unsupervised manner; the network discovers the underlying "height" of the material simply by learning to match its appearance from different angles.
  2. Neural Texture Pyramid: Similar to a standard mipmap, this component stores "latent features" rather than raw colors. It uses trilinear interpolation to retrieve features based on the required resolution.
  3. MLP Decoder: A compact, 4-layer fully-connected neural network that processes the interpolated features and lighting/view directions to output the final RGB reflectance value.

Key Insights and Innovations

  • Unsupervised Geometry Learning: The neural offset module learns to simulate 3D depth without explicit height maps. This allows the model to represent materials where a heightfield is not well-defined, such as fuzzy sweaters or complex fiber clusters.
  • Encoder-less Optimization: Rather than using a general encoder to compress data, NeuMIP optimizes texture features directly for each specific material. This specialization allows the final decoder to remain extremely small and fast.
  • Progressive Training: To avoid learning the noise inherent in Monte Carlo training data, the authors apply a Gaussian blur during training that progressively decays, ensuring the network learns smooth, coherent material properties.

Results and Performance

NeuMIP demonstrates significant improvements over previous benchmarks, such as the neural material compression technique by Rainer et al. [2020].

Qualitative comparison showing NeuMIP's ability to match reference path-traced materials across multiple resolutions with high fidelity and realistic parallax.
  • Efficiency: The model uses 11.5x fewer weight parameters and approximately 2.7x fewer texture channels than preceding state-of-the-art methods.
  • Accuracy: It consistently achieves lower error rates (MSE and LPIPS) across various materials, particularly those with significant non-flat geometry like cliff rocks.
  • Speed: Training for a new material is highly efficient, taking only 45 to 90 minutes on a single consumer-grade GPU (RTX 2080 Ti).

Limitations and Future Work

While NeuMIP excels at diffuse and moderately complex materials, the authors note that highly specular or "glinty" materials (like metallic car paint) remain a challenge. Future work may involve adding stochastic noise vectors to the decoder to synthesize these details. Additionally, the current framework focuses on reflectance (RGB) and could be extended to support semi-transparent materials by predicting alpha values.

Summary updated

Abstract

We propose NeuMIP, a neural method for representing and rendering a variety of material appearances at different scales. Classical prefiltering (mipmapping) methods work well on simple material properties such as diffuse color, but fail to generalize to normals, self-shadowing, fibers or more complex microstructures and reflectances. In this work, we generalize traditional mipmap pyramids to pyramids of neural textures, combined with a fully connected network. We also introduce neural offsets, a novel method which enables rendering materials with intricate parallax effects without any tessellation. This generalizes classical parallax mapping, but is trained without supervision by any explicit heightfield. Neural materials within our system support a 7-dimensional query, including position, incoming and outgoing direction, and the desired filter kernel size. The materials have small storage (on the order of standard mipmapping except with more texture channels), and can be integrated within common Monte-Carlo path tracing systems. We demonstrate our method on a variety of materials, resulting in complex appearance across levels of detail, with accurate parallax, self-shadowing, and other effects.
BibTeX
@article{kuznetsov2021neumip,
  author = {Alexandr Kuznetsov and Krishna Mullia and Zexiang Xu and Milos Hasan and Ravi Ramamoorthi},
  title = {{NeuMIP: multi-resolution neural materials}},
  journal = {ACM Transactions on Graphics},
  year = {2021},
  doi = {10.1145/3476576.3476763},
}

Loading related content...

Figure