Intrinsic Scene Properties From Hyperspectral Images and LiDAR

Xudong Jin, Yanfeng GuWorkshops @ CVPR 2019 · DOI

Introduction

Remote sensing has long relied on two complementary but distinct modalities: Hyperspectral Imaging (HSI) and Light Detection and Ranging (LiDAR). HSI provides rich spectral information across hundreds of narrow bands, enabling precise material identification. However, it often suffers from low spatial resolution. Conversely, LiDAR offers dense, high-resolution 3D geometric data but lacks the spectral depth to distinguish between materials with similar heights, such as vegetation and synthetic green surfaces.

A significant hurdle in fusing these data types is Intrinsic Image Decomposition (IID)—the task of separating an image into its fundamental components: reflectance (the "true" color of the material) and shading (the effects of illumination and geometry). While well-studied in computer vision for RGB images, IID in remote sensing is complicated by sub-pixel geometric complexity and the inherent data mismatch between structured HSI grids and unorganized LiDAR point clouds.

The IHSPC Framework

To address these challenges, researchers have introduced a framework that generates an Intrinsic Hyperspectral Point Cloud (IHSPC). Unlike traditional methods that flatten LiDAR data into a 2D Digital Surface Model (DSM), this approach operates directly on the raw 3D point cloud to preserve sub-pixel details.

The sub-pixel rendering model bridges the gap between structured HSI pixels and discrete LiDAR points by modeling sub-pixel reflectances and shapes.

The core of the methodology lies in a novel sub-pixel rendering model. By treating each HSI pixel as an aggregate of multiple sub-pixel shapes and materials derived from LiDAR points, the model can accurately account for how light interacts with complex 3D surfaces. This process utilizes Monte Carlo integration to bridge the gap between continuous physical light models and discrete point data. Furthermore, the framework employs L1-Graph coding as a sparsity prior, assuming that spatially proximal points likely share similar material properties, which aids in recovering consistent reflectance signatures.

Key Results and Applications

The resulting IHSPC is a high-fidelity 3D representation where each point is enriched with a full hyperspectral reflectance signature, stripped of shadows and lighting artifacts. This decomposition provides several advantages for downstream tasks.

Comparison of input HSI and LiDAR data with the resulting Intrinsic Hyperspectral Point Cloud (IHSPC) output.

In land-cover classification tests using the 2018 IEEE GRSS Data Fusion Contest dataset, the IHSPC achieved an Overall Accuracy (OA) of 80.1%, a substantial improvement over the 66.7% achieved by standard hyperspectral point clouds. This boost is largely attributed to the removal of "shadow noise," which typically causes spectrally identical materials to appear different under varying light conditions.

Beyond classification, the framework enables physics-based data augmentation and hyperspectral super-resolution. By understanding the underlying geometry and lighting, the system can "re-render" scenes from new perspectives or under different solar positions, providing a powerful tool for training deep learning models in data-scarce environments.

Conclusion and Future Outlook

The IHSPC framework represents a significant step forward in remote sensing data fusion. By moving away from 2D grid-based processing and embracing the inherent 3D nature of LiDAR, it achieves a more physically accurate decomposition of complex scenes.

While challenges remain—such as accounting for inter-reflections in dense urban environments and the high computational cost of point cloud operations—the ability to extract intrinsic material properties at high resolution opens new doors for environmental monitoring, urban planning, and more robust automated scene analysis.

Summary updated

BibTeX
@inproceedings{jin2019intrinsic,
  author = {Xudong Jin and Yanfeng Gu},
  title = {{Intrinsic Scene Properties From Hyperspectral Images and LiDAR}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00183},
}
Figure