Introduction
Inverse rendering is the foundational task of decomposing a 3D scene into its constituent geometry, material properties, and lighting conditions from 2D observations. This process is inherently ill-posed; for instance, a dark region in an image could represent a black object or a white object obscured by shadow. Traditional approaches relying solely on RGB data struggle with this ambiguity because color is a product of both intrinsic albedo and extrinsic illumination.
The LiDAR Advantage
While LiDAR is typically utilized for geometric depth, its intensity values—the power of the reflected laser pulse—offer a unique advantage. Unlike RGB cameras, LiDAR uses active infrared illumination, making its readings largely independent of ambient shadows or sunlight. InvRGB+L leverages this "active" signal to provide a consistent reference for material properties across varying lighting conditions, effectively acting as a "second opinion" to resolve visual ambiguities.
Methodology: Unified Reflectance Modeling
The framework utilizes a Dynamic 3D Gaussian Splatting (3DGS) Scene Graph to represent complex urban environments, handling both static backgrounds and moving objects. To bridge the gap between RGB and LiDAR, the authors introduce several key innovations:
- Physics-based LiDAR Reflectance Model: Moving beyond simple Lambertian assumptions, the system employs a Cook-Torrance BRDF model for LiDAR. This accounts for surface roughness and specularity, which is crucial for modeling metallic or wet surfaces in the infrared spectrum.
- RGB-LiDAR Albedo Consistency Loss: This loss function enforces spatial correlation between RGB and LiDAR albedo. While the absolute values differ across spectra, their gradients and neighborhoods should behave similarly as they originate from the same physical surface microstructure.
- Differentiable PBR Pipeline: The system uses a physically-based rendering approach to optimize scene parameters, accounting for sky dome lighting and primary sun sources to produce realistic shadows.
Results and Evaluation
Evaluated on the Waymo Open Dataset and custom sequences, InvRGB+L achieves state-of-the-art performance in several categories:
- Relighting Accuracy: The model reached a 30.42 PSNR in relighting tasks, significantly outperforming previous benchmarks like UrbanIR (28.84 PSNR).
- LiDAR Simulation: The model's ability to simulate LiDAR intensity resulted in a lower RMSE (0.063) compared to specialized simulation baselines like LiDARsim and NFL.
- Qualitative Fidelity: The framework successfully removes "baked-in" shadows from road textures and enables photorealistic "night-to-day" transitions and dynamic object insertion.
Conclusion
InvRGB+L demonstrates that fusing active LiDAR intensity with passive RGB video significantly improves the disentanglement of lighting and materials. While challenges remain in modeling complex nighttime environments with multiple point lights and improving computational efficiency, this work sets a new standard for high-fidelity, relightable 3D scene reconstruction.
Summary updated