Cross-Spectral Gaussian Splatting with Spatial Occupancy Consistency

Haipeng Guo, Huanyu Liu, Jiazheng Wen, Junbao LiAAAI 2025 · DOI

Sure, here's a blog-style summary of the paper "Cross-Spectral Gaussian Splatting with Spatial Occupancy Consistency":

TL;DR

This paper introduces Cross-Spectral Gaussian Splatting with Spatial Occupancy Consistency (SOC-GS), a novel method for creating 3D scene representations from images captured by cameras with different spectral sensitivities (e.g., RGB, infrared, multi-spectral). SOC-GS achieves more accurate cross-spectral alignment and real-time rendering by jointly optimizing Gaussian splatting representations and camera poses, overcoming limitations of previous approaches.

What Problem Does This Paper Address?

Capturing a complete view of a scene using a single type of camera can be challenging. Different light spectrum sensitivities offer complementary information. For instance, infrared cameras can see through fog or darkness, while multi-spectral cameras can capture detailed material properties. However, integrating data from these diverse sensors into a unified 3D scene representation is difficult due to issues like:

  • Cross-spectral misalignment: Images from different cameras might not perfectly align, leading to blurry or inaccurate 3D reconstructions.
  • Camera pose estimation: Accurately determining the relative positions and orientations (poses) of cameras with different spectral sensitivities is a significant hurdle. Traditional Structure-from-Motion (SfM) methods often struggle with cross-spectral data.
  • Real-time rendering and scalability: Existing methods for cross-spectral scene representation can be computationally expensive, making real-time rendering and handling large scenes difficult.

This paper aims to address these challenges by developing a method that can effectively represent and render 3D scenes from cross-spectral images, even when camera poses are not perfectly known or synchronized.

Methods

The proposed method, SOC-GS, builds upon the recent advancements in 3D Gaussian Splatting (3DGS). Here's a breakdown of its key techniques:

  • Shared Gaussian Attributes: The core idea is to represent the 3D scene using Gaussian primitives. Crucially, the fundamental attributes of these Gaussians (mean, covariance, and opacity) are shared across all spectral views. This enforces spatial consistency.
  • Independent Spectral Colors: While the geometry is shared, the color for each spectral view is represented independently using spherical harmonic coefficients. This allows each spectrum to have its own unique color representation.
  • Two-Stage Cross-Spectral Pose Initialization: To tackle the pose estimation problem, SOC-GS employs a novel two-stage approach:
  • Initial Pose Estimation: It first uses a pre-trained keypoint matcher (LoFTR) and the initial Gaussian pointcloud to roughly estimate the poses of the cross-spectral cameras. This leverages the spatial occupancy consistency of the Gaussians.
  • Joint Differentiable Optimization: These initial poses then serve as input for a joint optimization process where both the Gaussian attributes and the camera poses are refined simultaneously. This ensures tighter alignment.
  • Cross-Spectral Adaptive Densify Controller: To handle differences in field-of-view (FOV) between cameras and improve reconstruction quality, an adaptive densification strategy is introduced. This controller helps fill in areas that might be missed by one spectral view but captured by another, particularly in non-overlapping regions.

Datasets

The paper evaluates SOC-GS on two datasets:

  • X-NeRF Dataset: This dataset contains 16 indoor scenes with synchronized RGB, multi-spectral, and infrared camera captures. The cameras are fixed, providing a baseline for comparison.
  • RealSense Dataset: The authors collected their own dataset using an Intel RealSense D435 depth camera, capturing various indoor and outdoor scenes with RGB and infrared (stereo) images. This dataset is designed to test the method's robustness in scenarios with less strict synchronization and wider variations.

Key Insights and Ideas

  • Spatial Occupancy Consistency is Key: The paper's central insight is that by enforcing consistent spatial occupancy for Gaussian primitives across different spectra, strong geometric alignment can be achieved. This shared geometric foundation is more robust than relying solely on color or texture matching.
  • Two-Stage Pose Optimization is Effective: The two-stage pose initialization and joint optimization pipeline proves crucial. It allows for a good initial guess that significantly speeds up and improves the accuracy of the final pose estimation and scene representation, especially when dealing with challenging cross-spectral data.
  • Adaptive Densification Bridges FOV Gaps: The adaptive densification controller is vital for handling real-world scenarios where cameras might have different fields of view. It ensures that information from all spectra is utilized to create a complete and high-quality representation.
  • 3DGS as a Foundation: Leveraging the efficiency and expressiveness of 3D Gaussian Splatting provides a strong base for real-time rendering and rapid optimization, which are significant improvements over previous implicit neural representation methods.

Key Results

  • Superior NVS Performance: SOC-GS consistently outperforms competing methods like X-NeRF in novel view synthesis (NVS) quality, achieving higher PSNR and SSIM scores across various spectral combinations on both X-NeRF and RealSense datasets.
  • Accurate Pose Estimation: The method demonstrates significantly better pose estimation accuracy (lower EM scores) compared to methods that rely solely on SfM or other pose estimation techniques, especially in cross-spectral scenarios.
  • Real-Time Rendering: SOC-GS achieves impressive rendering speeds, reaching up to 120 FPS at 1Mpx resolution, which is substantially faster than X-NeRF.
  • Improved Cross-Spectral Alignment: Qualitative results show that SOC-GS produces more tightly aligned and detailed renderings, with depth maps that align better with the visual output, indicating a more accurate geometric understanding of the scene.

Limitations and Future Work

  • Reliance on Camera Intrinsics: The method assumes that camera intrinsics (focal length, principal point, distortion) are known and calibrated.
  • Initialization Sensitivity: While the two-stage approach mitigates this, the initial pose estimation can still be sensitive to the quality of matching keypoints.
  • Future Work: The authors suggest further exploring the benefits of cross-spectral views for Gaussian geometry optimization to enhance scene geometry representation accuracy. They also aim to extend the method to more complex scenarios and potentially reduce reliance on pre-calibrated intrinsics.

Summary updated

BibTeX
@inproceedings{guo2025cross,
  author = {Haipeng Guo and Huanyu Liu and Jiazheng Wen and Junbao Li},
  title = {{Cross-Spectral Gaussian Splatting with Spatial Occupancy Consistency}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i3.32333},
}
Figure