TensoFlow: Tensorial Flow-based Sampler for Inverse Rendering

Chun Gu, Xiaofei Wei, Li Zhang, Xiatian Zhu

Abstract Inverse rendering aims to recover scene geometry, material properties, and lighting from multi-view images. Given the complexity of light-surface interactions, importance sampling is essential for the evaluation of the rendering equation, as it reduces variance and enhances the efficiency of Monte Carlo sampling. Existing inverse rendering methods typically use pre-defined non-learnable importance samplers in prior manually, struggling to effectively match the spatially and directionally varied integrand and resulting in high variance and suboptimal performance. To address this limitation, we propose the concept of learning a spatially and directionally aware importance sampler for the rendering equation to accurately and flexibly capture the unconstrained complexity of a typical scene. We further formulate TensoFlow, a generic approach for sampler learning in inverse rendering, enabling to closely match the integrand of the rendering equation spatially and directionally. Concretely, our sampler is parameterized by normalizing flows, allowing both directional sampling of incident light and probability density function (PDF) inference. To capture the characteristics of the sampler spatially, we learn a tensorial representation of the scene space, which imposes spatial conditions, together with reflected direction, leading to spatially and directionally aware sampling distributions. Our model can be optimized by minimizing the difference between the integrand and our normalizing flow. Extensive experiments validate the superiority of TensoFlow over prior alternatives on both synthetic and real-world benchmarks.

Introduction

Inverse rendering is often considered the "holy grail" of 3D computer vision. The objective is to decompose a set of 2D images into their constituent 3D components: geometry, material properties (such as albedo, metallicness, and roughness), and scene illumination. At the center of this challenge is the Rendering Equation, an integral that describes how light reflects off a surface point toward a camera. Because this integral is analytically intractable for complex scenes, researchers rely on Monte Carlo sampling—averaging multiple random samples of incident light directions.

The efficiency of this process depends heavily on importance sampling. Traditional methods employ fixed, non-learnable samplers, such as cosine-weighted or GGX distributions. However, these samplers are "blind" to the specific spatial and directional nuances of a scene, such as the location of bright light sources or the presence of occluding geometry. This mismatch between the sampler and the actual light distribution leads to high variance, manifesting as visual noise and requiring a prohibitive number of samples to achieve clean results. TensoFlow addresses this by introducing a learnable, spatially and directionally aware importance sampler.

Methodology

TensoFlow replaces static sampling formulas with a neural distribution that "learns" the lighting structure of the scene. The framework consists of several key components:

  1. Normalizing Flows: The core of the sampler is a Normalizing Flow (specifically based on the NICE architecture). This mathematical framework transforms a simple uniform distribution into a complex, multi-modal distribution that can accurately match the incident light at any surface point.
  2. Tensorial Priors: To achieve spatial awareness, TensoFlow utilizes a Vector-Matrix tensorial encoder. This allows the model to query local features based on the 3D coordinates ($x, y, z$) of a surface point, ensuring the sampler adapts to local geometry and shadows.
  3. Directional Awareness: The sampler is conditioned on the reflected direction ($\omega_r$). This is vital because specular highlights and reflections are highly dependent on the viewing angle.
  4. Piecewise-Quadratic Coupling: To model sharp highlights and complex reflections, the flow uses piecewise-quadratic functions. This provides the necessary expressiveness while maintaining mathematical invertibility for both sampling and PDF inference.
Overview of the TensoFlow pipeline and the Tensorial Normalizing Flow architecture.

The sampler is optimized by minimizing the cross-entropy loss between its predicted distribution and the actual integrand of the rendering equation. In effect, the sampler learns to place its samples where the light is most significant, drastically reducing variance.

Experiments and Results

The authors evaluated TensoFlow on two major benchmarks: the TensoSDF Synthetic Dataset and the real-world Stanford-ORB dataset. The synthetic dataset, featuring objects like a Rover and a Dragon, allowed for precise measurement against ground-truth data.

TensoFlow consistently outperformed state-of-the-art methods such as NeRO and TensoSDF. On synthetic benchmarks, TensoFlow achieved an average PSNR of 26.933, compared to 25.971 for TensoSDF and 23.204 for NeRO. Beyond image reconstruction quality, the method demonstrated a massive reduction in sampling variance—approximately 3x lower than traditional GGX samplers.

Qualitative comparison of TensoFlow against state-of-the-art methods NeRO and TensoSDF on synthetic benchmarks.

Qualitatively, the reduced variance leads to significantly cleaner material decomposition. The recovered albedo, metallic, and roughness maps are sharper and more accurate because the optimization process is not hampered by the noise inherent in fixed samplers.

Discussion and Conclusion

The primary insight of TensoFlow is that importance sampling should not be a static preprocessing step but a learnable component of the inverse rendering pipeline. By giving the sampler "memory" of the scene's spatial and directional lighting characteristics, TensoFlow achieves superior disentanglement of light and materials.

While the use of normalizing flows introduces some computational overhead compared to simple formulas, the gains in efficiency and accuracy are substantial. Future work may explore extending this learnable sampler approach to more complex light transport phenomena, such as volumetric rendering or sub-surface scattering, further pushing the boundaries of high-fidelity scene reconstruction.

Summary updated

Abstract

Inverse rendering aims to recover scene geometry, material properties, and lighting from multi-view images. Given the complexity of light-surface interactions, importance sampling is essential for the evaluation of the rendering equation, as it reduces variance and enhances the efficiency of Monte Carlo sampling. Existing inverse rendering methods typically use pre-defined non-learnable importance samplers in prior manually, struggling to effectively match the spatially and directionally varied integrand and resulting in high variance and suboptimal performance. To address this limitation, we propose the concept of learning a spatially and directionally aware importance sampler for the rendering equation to accurately and flexibly capture the unconstrained complexity of a typical scene. We further formulate TensoFlow, a generic approach for sampler learning in inverse rendering, enabling to closely match the integrand of the rendering equation spatially and directionally. Concretely, our sampler is parameterized by normalizing flows, allowing both directional sampling of incident light and probability density function (PDF) inference. To capture the characteristics of the sampler spatially, we learn a tensorial representation of the scene space, which imposes spatial conditions, together with reflected direction, leading to spatially and directionally aware sampling distributions. Our model can be optimized by minimizing the difference between the integrand and our normalizing flow. Extensive experiments validate the superiority of TensoFlow over prior alternatives on both synthetic and real-world benchmarks.
BibTeX
@inproceedings{gu2025tensoflow,
  author = {Chun Gu and Xiaofei Wei and Li Zhang and Xiatian Zhu},
  title = {{TensoFlow: Tensorial Flow-based Sampler for Inverse Rendering}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00055},
}
Figure