Geometry-Aware Global Feature Aggregation for Real-Time Indirect Illumination

Meng Gai, Guoping Wang, Sheng Li

TL;DR

This paper introduces a novel neural network architecture for real-time global illumination that predicts indirect lighting in screen space. By combining a geometry-aware attention mechanism with a "monochromatic" design that processes color channels independently, the model achieves high-quality HDR results in just 12ms, effectively capturing long-range light transport and realistic color bleeding.

What Problem Does This Paper Address?

Global Illumination (GI)—the way light bounces off surfaces to illuminate others—is essential for visual realism, especially in VR/AR. However, calculating these "indirect" bounces in real-time is notoriously difficult. Traditional methods like Monte Carlo path tracing are too slow, while existing real-time approximations often miss subtle effects like color bleeding.

Recent deep learning approaches have tried to predict GI, but they face three major hurdles:

  1. Limited Receptive Fields: Standard Convolutional Neural Networks (CNNs) look at local neighborhoods. This makes it hard for them to "see" light bouncing from a distant wall to the current pixel.
  2. Color Inconsistency: Generative models often "average out" colors from training data, leading to muddy results that don't respect the physical relationship between light and surface color.
  3. Linearity and HDR: Most models work in Low Dynamic Range (LDR), losing the physical linearity required for realistic lighting changes (e.g., if you double the light source intensity, the indirect light should also double).

Methods

The authors propose a "neural solver" inspired by the physics of the rendering equation. Their architecture features two standout innovations:

  • Geometry-Aware Global Feature Aggregation (GFA): Instead of relying on deep stacks of convolutions to expand the field of view, they use a modified multi-head attention mechanism. Crucially, the attention weights are guided by geometry features (positions, normals, and depth). This acts as a learnable version of the "geometry term" in physics, allowing the network to calculate how much light one point in the image contributes to another based on their distance and orientation.
  • Monochromatic Shading Generator: In a move that defies standard deep learning intuition, the network processes each RGB channel independently using a shared "shading" network. Since light transport is wavelength-independent (physics-wise), this design eliminates redundant inter-channel parameters. It forces the network to learn the logic of light transport rather than memorizing color patterns, which significantly improves generalization to new, colored lighting setups.
  • HDR-Native Training: The model is trained directly on linear High Dynamic Range data. They use an exponential activation function and "exposure augmentation"—randomly scaling the brightness during training—to ensure the model understands that lighting is a linear process.

Datasets

The researchers found existing datasets lacking for this specific task, so they created a new one:

  • 3D-Front HDR Dataset: Based on the 3D-Front indoor layout dataset, they rendered over 30,000 HDR images across 1,500 layouts using Blender Cycles.
  • Rich Annotations: Unlike previous datasets, this one includes pixel-wise factorized data: direct vs. indirect lighting, diffuse vs. specular components, albedo, surface normals, and depth. This allows for "fine-grained supervision," where the network can focus specifically on learning the indirect shading component.

Key Insights and Ideas

  • Attention as Physics: The "aha moment" here is treating the attention mechanism not just as a global context tool, but as a direct proxy for the integral in the rendering equation. By using geometry to guide attention, the network learns visibility and distance decay naturally.
  • The Power of Independence: By forcing the network to be monochromatic, the authors solved the "color-shifting" problem common in other GI networks. Because the network can't "see" other channels, it can't accidentally mix them, ensuring that a red light bounce stays red and doesn't turn into a generic gray average.
  • Demodulating Shading: The network doesn't predict the final image; it predicts the shading (the light hitting the surface). This is then multiplied by the surface's texture (albedo) in a post-process. This "reflectance factorization" makes the network's job much easier because shading is generally smoother and lower-frequency than complex textures.

Key Results

  • Performance: The model runs in approximately 12ms on an NVIDIA RTX 3090 at 768x512 resolution, making it viable for real-time applications.
  • Quality: It significantly outperformed previous state-of-the-art methods (like Pix2pix and Xin et al.) across all metrics. Specifically, it achieved a PSNR of 31.88 (vs. 30.28 for the next best) and a much lower LPIPS score of 0.0203, indicating results that are much closer to ground truth in human perception.
  • Generalization: The model successfully handled "unseen" scenes with complex colored lights and environment maps, capturing long-range effects (like a bright wall illuminating a distant corner) that CNN-based models missed entirely.

Limitations and Future Work

  • Screen-Space Constraints: Like all screen-space methods, the model can't account for light sources or geometry that are "behind the camera" or off-screen.
  • Glossy Surfaces: The current implementation focuses on diffuse (matte) reflections. Handling highly specular or mirror-like surfaces remains a challenge for future iterations.
  • Temporal Stability: While the results look great in stills, the authors note that drastic camera movements can cause flickering. They suggest integrating Temporal Anti-Aliasing (TAA) to smooth this out.
  • Mobile VR: While 12ms is fast, the authors aim to further optimize the architecture for mobile VR headsets (like Meta Quest) using techniques like bilateral up-sampling.

Summary updated

Abstract

Real-time rendering with global illumination is crucial to afford the user realistic experience in virtual environments. We present a learning-based estimator to predict diffuse indirect illumination in screen space, which then is combined with direct illumination to synthesize globally-illuminated high dynamic range (HDR) results. Our approach tackles the challenges of capturing long-range/long-distance indirect illumination when employing neural networks and is generalized to handle complex lighting and scenarios. From the neural network thinking of the solver to the rendering equation, we present a novel network architecture to predict indirect illumination. Our network is equipped with a modified attention mechanism that aggregates global information guided by spacial geometry features, as well as a monochromatic design that encodes each color channel individually. We conducted extensive evaluations, and the experimental results demonstrate our superiority over previous learning-based techniques. Our approach excels at handling complex lighting such as varying-colored lighting and environment lighting. It can successfully capture distant indirect illumination and simulates the interreflections between textured surfaces well (i.e., color bleeding effects); it can also effectively handle new scenes that are not present in the training dataset.
BibTeX
@misc{gai2025geometry,
  author = {Meng Gai and Guoping Wang and Sheng Li},
  title = {{Geometry-Aware Global Feature Aggregation for Real-Time Indirect Illumination}},
  year = {2025},
  eprint = {2508.08826},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.08826},
}
Figure