This paper from the University of Cambridge presents a significant leap in how we represent and generate complex real-world materials. By moving from massive, per-pixel data structures to a "one code per material" approach, the authors enable high-fidelity material reconstruction and the generation of entirely new, physically grounded textures.
TL;DR
The authors introduce a neural representation for Bidirectional Texture Functions (BTFs) that compresses a complex material's appearance into a single 256-float code. This compact approach not only sets new state-of-the-art benchmarks for reconstruction quality but also enables a generative flow model to sample entirely new, relightable materials from a learned latent space.
What Problem Does This Paper Address?
Capturing the "soul" of a material—the way light dances across carpet loops, shadows fall into leather cracks, or highlights shift with parallax—requires Bidirectional Texture Functions (BTFs). Unlike standard analytic models (SVBRDFs) used in most game engines, BTFs capture complex effects like self-shadowing and inter-reflections by measuring a sample under thousands of light and view combinations.
The challenge is twofold:
- Size: A single measured material can consist of billions of values, reaching gigabytes in size.
- Generativity: Existing neural BTF representations use "per-texel latent maps" (essentially images of features). While these compress the data, they don't allow you to easily "sample" a new material. You can't just pick a random feature map and expect it to look like a coherent, realistic material.
Methods
The researchers propose a two-tier neural architecture to solve these issues:
- Per-Material Coordinate Network: For maximum accuracy, they fit a Fourier-feature MLP to a single material. This network maps a texel position $(x)$ and a light/view direction pair (represented by Rusinkiewicz vectors) directly to an RGB value.
- The Compact Model: To enable a shared space, they replace the per-material network with a single, massive Shared Decoder ($g_\theta$) conditioned on a global 256-float code ($z_m$). This reduces the material-specific data from millions of features to just 256 numbers.
- Set Encoder: To find the right code for a material, they use a permutation-invariant encoder that processes a small set of random samples from a measurement.
- Rectified Flow for Generation: Because every material is now just a point in a 256-dimensional space, the authors can train a class-conditional rectified flow model to sample new points. These sampled codes can then be "rendered" by the shared decoder into full, relightable BTFs.
- Colourway Transfer: To combat data scarcity, they developed a method to transfer the color characteristics of one material onto the physical structure of another within the same class, effectively augmenting their training set.
Datasets
- UBO2014: The gold standard for measured BTFs, containing 84 materials across seven classes (carpet, fabric, felt, leather, stone, wallpaper, and wood).
- Augmented Dataset: Using their "colourway" technique and hue rotations, the authors expanded the 84 base captures into a dataset of 886 materials to provide enough variety for the generative flow model to learn.
Key Insights and Ideas
- Global vs. Local Latents: The "aha moment" is realizing that a material's identity can be captured globally. Moving from per-texel maps to a single code treats a material as a single entity in a vector space, which is the prerequisite for true generative modeling.
- Measured Realism in Generative Space: Most material generators learn from flat photos or synthetic data. This model learns from measured light transport. When the model "hallucinates" a new fabric, it isn't just dreaming up a pattern; it's generating a structure that inherently understands how to cast shadows on itself.
- The Kilobyte Scale: The compact model brings measured BTFs down to the kilobyte scale per material, making them practical for complex scenes with hundreds of unique surfaces.
Key Results
- Superior Reconstruction: The per-material model outperforms the previous state-of-the-art (Xia et al.) by +3.2 dB PSNR. Even the highly compressed compact model beats prior per-texel methods (like Rainer et al.) by +1.2 dB PSNR while using 14x fewer values per material.
- Rate-Distortion Efficiency: As shown in Figure 4, the compact model achieves a much better balance between storage size and visual quality than any previous baseline.
- Generative Quality: The flow model achieved the lowest Kernel Inception Distance (KID) and Maximum Mean Discrepancy (MMD) compared to image-based generators, proving that it produces more realistic and diverse samples.
Limitations and Future Work
- Structural Novelty: Because the model is trained on only 84 unique physical structures, its generative "creativity" is mostly limited to new color combinations and subtle blends. It doesn't yet "invent" entirely new weave patterns.
- Bake-out Time: To use these materials in a real-time renderer, the neural model must be decoded into a light-view table, which currently takes about 3.7 minutes on an NVIDIA A100.
- Tiling: The model represents a fixed-size measured patch. Future work could integrate tiling and synthesis techniques to cover larger surfaces.
The authors hope this work makes measured appearance a practical target for generative modeling, bringing us one step closer to digital environments that are indistinguishable from reality.
Summary updated