Introduction
Recovering the lighting of a scene from a photograph is a fundamental yet highly ill-posed inverse rendering problem. For applications like Augmented Reality (AR), knowing the exact sun direction is critical for rendering realistic shadows and highlights that match the real-world environment. While deep learning has significantly advanced single-image lighting estimation, these methods often lack robustness when visual cues are sparse—for example, when a frame is entirely in shadow or lacks clear geometric references.
To address these limitations, researchers have turned to video sequences to aggregate information across multiple viewpoints. However, previous multi-image approaches often relied on complex, handcrafted statistical post-processing, such as mean-shift clustering and manual outlier removal. This paper introduces an end-to-end Transformer-based architecture that replaces these clunky heuristics with a purely attention-driven mechanism, achieving state-of-the-art accuracy in sun direction prediction.
The Proposed Architecture
The authors propose a hybrid CNN-Transformer pipeline designed to estimate the 3D Cartesian vector of the sun from a sequence of images. The process begins with Patch Extraction, where subimages are randomly cropped from video frames. A pre-trained ResNet18 backbone then extracts feature embeddings from these patches.
These embeddings are processed through a Spatio-Temporal Attention module. By using a Transformer network, the system inherently learns to weigh reliable patches—such as those containing clear cast shadows or sky gradients—while ignoring noisy or uninformative ones. Finally, the network outputs sun direction estimates in local camera coordinates, which are then calibrated into a unified global coordinate system using ego-motion data (from Structure-from-Motion) and averaged for a stable final result.
Orientation-Invariant Positional Encoding
A key contribution of this work is the Orientation-Invariant Positional Encoding. To allow the Transformer to understand the spatial and temporal relationships between random crops, the authors developed a custom 3D cyclic positional encoding.
This encoding captures the 2D angular coordinates of each crop relative to the camera's field of view and the temporal camera yaw angle derived from vehicle motion. By utilizing nested trigonometric functions (sin∘sin and sin∘cos), the encoding remains unique yet periodic, ensuring that the transition from 359° to 0° is handled smoothly without discontinuities. This allows the model to remain invariant to specific imaging parameters like aspect ratio or focal length.
Experimental Results and Key Insights
The method was evaluated on two major datasets: KITTI (real-world autonomous driving sequences) and SUN360 (panoramic images adapted into synthetic perspective sequences).
Performance Metrics
- SUN360 Dataset: The proposed method achieved a median angular error of 22.53°, significantly outperforming the previous state-of-the-art mean-shift aggregation method (31.66°).
- KITTI Dataset: The mean angular error was reduced to 8.76°. Notably, the maximum error dropped from 45.93° to 25.97°, demonstrating the Transformer's superior ability to suppress extreme outliers that typically plague statistical methods.
Qualitative Impact
In AR applications, the Transformer-aggregated lighting produced highly stable, temporally coherent shadows. In contrast, single-image estimates often resulted in "jittery" shadows that fluctuated wildly between frames. The joint prediction mechanism of the Transformer naturally forces individual patch estimates to cluster tightly around the true mean, providing a consensus that handcrafted algorithms struggle to achieve.
Conclusion and Future Work
This research demonstrates that attention mechanisms can effectively replace handcrafted statistical filtering for multi-frame lighting estimation. By providing proper spatial and temporal context through novel positional encodings, the model becomes highly resilient to localized lack of cues.
Future work aims to expand the model's scope beyond sun direction to include factors like cloudiness and exposure. Additionally, the authors suggest that the 3D scene lighting data could be integrated into SLAM (Simultaneous Localization and Mapping) systems to help initialize global camera orientation or improve shadow removal algorithms.
Summary updated