Exploring Multi-modal Neural Scene Representations With Applications on Thermal Imaging

Mert Özer, Maximilian Weiherer, Martin Hundhausen, Bernhard Egger

Abstract

Neural Radiance Fields (NeRFs) have revolutionized 3D scene reconstruction and novel view synthesis using RGB imagery. However, extending these representations to non-visible modalities, such as thermal (far-infrared) or near-infrared (NIR) imaging, presents unique challenges including feature scarcity and geometric ambiguity. This paper systematically evaluates architectural strategies for multi-modal NeRF integration. By introducing ThermalMix, a high-quality aligned dataset of RGB and thermal image pairs, we demonstrate that a dual-branch architecture (RGB-X) with a shared density network outperforms traditional fine-tuning or independent training. Our findings show that thermal data can effectively regularize scene geometry, particularly in 360-degree captures where thermal backgrounds are often featureless.

Introduction

Neural Radiance Fields (NeRFs) have become the gold standard for synthesizing novel views of a scene using RGB images. However, many real-world applications—ranging from medical imaging and agriculture to defense—rely on multi-modal data like thermal or near-infrared (NIR) sensors. Integrating these modalities into NeRFs is difficult for two primary reasons:

  1. Feature Scarcity: Thermal images are often blurry and lack the high-frequency textures found in RGB data. Standard computer vision tools, such as COLMAP, often fail to estimate camera poses from thermal data alone because they cannot identify sufficient matching points between views.
  2. Geometric Discrepancy: The perceived "geometry" of a scene can vary across sensors. For instance, a heat source might appear as a glowing, indistinct blob that does not align perfectly with the physical edges visible in an RGB image.

This research seeks to identify the optimal architectural strategy to fuse these disparate data types into a single, coherent 3D representation.

Methodology

Using Instant-NGP as the base model for its speed and efficiency, the researchers compared four distinct strategies for incorporating a second modality ("X"):

  • Training from Scratch (TS): This baseline approach involves training two entirely separate NeRF models—one for RGB and one for the second modality.
  • Fine-tuning (FT): A model is pre-trained on RGB data and subsequently fine-tuned on the second modality, assuming the underlying geometry is identical across both.
  • Second Branch (RGB-X): A single density network feeds into two separate "color" branches. During training, gradients from both the RGB and the second modality flow back into the shared density network, allowing both to influence the learned geometry.
  • Separate Component (SC): Similar to the second branch, but the second modality's branch is restricted from influencing the density network. It must learn to represent the second modality using geometry derived solely from RGB data.

To ensure the images were perfectly aligned for training, the authors developed a custom cross-modality calibration technique. They utilized a perforated aluminum plate; by heating or cooling the plate, the holes became visible to both thermal and RGB sensors, allowing for precise geometric alignment and pose estimation.

The ThermalMix Dataset

To evaluate these strategies, the researchers introduced ThermalMix, a newly captured dataset consisting of six common objects: Face, Hand, Panel, Lion, Pan, and Laptop. The dataset includes approximately 360 high-quality aligned RGB and thermal image pairs, covering both forward-facing and 360-degree perspectives. Additionally, the authors utilized an existing multi-sensor dataset containing RGB, NIR, and depth maps to test the generalization of their findings.

Key Insights and Results

The Power of Shared Density

The RGB-X strategy emerged as the superior approach. The researchers found that allowing thermal data to influence the density network actually helped the model regularize the scene better than using RGB data alone. This shared learning process allows the strengths of each modality to compensate for the weaknesses of the other.

Addressing Thermal Ambiguity

In 360-degree thermal captures, backgrounds often appear static and uniform, which typically confuses standard NeRFs and leads to "floaters" (artifacts in empty space). By using RGB data to guide the geometry—as seen in both the RGB-X and SC strategies—the model effectively eliminates these artifacts, resulting in much cleaner reconstructions.

Quantitative Performance

In 360-degree scenes, the RGB-X strategy significantly outperformed the baseline (TS), achieving a PSNR of 28.49 compared to the baseline's 21.81. While the baseline remains slightly superior for pure RGB reconstruction (due to the lack of "interference" from thermal gradients), the RGB-X and SC strategies yielded highly compelling results, proving that a single model can represent both modalities without a significant loss in quality.

Conclusion and Future Work

This study confirms that the "Second Branch" (RGB-X) approach is a robust choice for general multi-modal neural scene representation, particularly for challenging modalities like thermal imaging. The primary limitation of the current work is its reliance on offline calibration using specialized hardware (the perforated plate). Future research should focus on "online" or learning-based calibration schemes that can automatically align different modalities during the training process, enabling the use of these techniques on uncalibrated, "in-the-wild" datasets.

Summary updated

BibTeX
@inproceedings{ozer2024exploring,
  author = {Mert Özer and Maximilian Weiherer and Martin Hundhausen and Bernhard Egger},
  title = {{Exploring Multi-modal Neural Scene Representations With Applications on Thermal Imaging}},
  booktitle = {Workshops @ ECCV},
  year = {2024},
  doi = {10.1007/978-3-031-92805-5\_6},
}
Figure