Abstract
Inverse rendering is a fundamental challenge in computer vision, requiring the decomposition of 2D images into 3D geometry, material properties, and lighting environments. A critical bottleneck in this process is the efficient evaluation of the rendering equation, which typically relies on Monte Carlo sampling. We present TensoFlow, a novel framework that replaces traditional, static importance samplers with a learnable, spatially-aware distribution. By utilizing normalizing flows conditioned on tensorial scene features, TensoFlow learns to sample light directions optimally for any specific point in a 3D scene. Our results demonstrate that this adaptive approach significantly reduces rendering noise and improves the accuracy of recovered scene parameters, outperforming state-of-the-art methods on both synthetic and real-world datasets.
Introduction
The goal of inverse rendering is to reconstruct a digital twin of a physical scene from a set of photographs. This involves solving the rendering equation, an integral that describes the light reflected from a surface point toward the camera. Because this integral lacks a closed-form solution for complex scenes, researchers employ Monte Carlo integration, which approximates the integral by averaging samples of incident light directions.
To improve efficiency, importance sampling is used to focus samples on directions that contribute most to the integral. However, standard methods often rely on fixed analytical distributions (e.g., GGX) that are independent of the local lighting environment. In practice, the optimal sampling distribution varies dramatically across a scene—a point under a direct spotlight requires a different sampling strategy than a point in a shadowed corner. TensoFlow addresses this limitation by introducing a learnable sampler that adapts to the local spatial and directional context.
Methodology
TensoFlow integrates a neural importance sampler into the inverse rendering pipeline. The core of the system is a normalizing flow architecture that transforms a simple base distribution into a complex, multi-modal distribution representing the incident light.
Tensorial Normalizing Flows
The sampler uses a normalizing flow based on the NICE architecture, which allows for both efficient sampling and exact probability density function (PDF) calculation. To make the sampler "spatially aware," it is conditioned on local scene features extracted from a Vector-Matrix (VM) tensorial representation. This compact representation allows the model to store and retrieve high-dimensional scene information without the memory overhead of dense 3D grids.
As shown in Figure 3, the system takes a ray-surface intersection point $x$ and queries the tensorial encoders $V_f(x)$ and $V_m(x)$. These features, along with the reflected direction $\omega_r$, condition the normalizing flow. This ensures that the sampler prioritizes directions that are both physically likely (based on material properties) and environmentally significant (based on local lighting).
Expressive Coupling Layers
To capture the complex, often jagged distributions of real-world lighting, TensoFlow employs piecewise-quadratic coupling functions. Unlike simple linear transformations, these functions provide the necessary expressiveness to model sharp peaks and intricate patterns in the light field. The model also utilizes half-vector modeling, which aligns the sampling distribution with the microfacet theory of material reflection, leading to more stable training and better results for specular surfaces.
Experiments and Results
We evaluated TensoFlow against state-of-the-art inverse rendering methods, including NeRO and TensoSDF, using two primary benchmarks: the synthetic TensoSDF dataset and the real-world Stanford-ORB dataset.
Quantitative Performance
On the TensoSDF synthetic dataset, which includes complex objects like a rover and a dragon, TensoFlow achieved a significant improvement in reconstruction quality. Our method reached an average Peak Signal-to-Noise Ratio (PSNR) of 26.933, compared to 25.971 for the baseline TensoSDF. This improvement stems from the sampler's ability to focus on high-contribution light directions, thereby reducing the variance of the Monte Carlo estimator.
Qualitative Analysis and Relighting
The benefits of the learnable sampler are most evident in the quality of the recovered material maps and the resulting relighting performance.
As illustrated in Figure 6, TensoFlow produces cleaner albedo, roughness, and metallic maps. The "Variance (Ours)" column highlights the dramatic reduction in speckle noise during the rendering process compared to traditional importance sampling. This leads to more realistic relighting results when the reconstructed objects are placed in novel lighting environments, as seen in the comparisons with NeRO and TensoSDF.
Conclusion and Future Work
TensoFlow represents a shift from hardcoded sampling heuristics to learned, data-driven distributions for inverse rendering. By combining normalizing flows with tensorial scene representations, we achieve a sampler that is both highly expressive and spatially aware. While the current implementation introduces a computational overhead due to the neural network queries required for each sample, it sets a new standard for accuracy and noise reduction in scene reconstruction. Future research will focus on optimizing the flow architecture for real-time performance and exploring end-to-end training pipelines that integrate geometry estimation with the learnable sampler.
Summary updated