Introduction
Neural Radiance Fields (NeRFs) have fundamentally changed the landscape of 3D scene reconstruction, enabling the generation of high-fidelity novel views from a set of 2D images. However, a significant limitation of standard NeRF implementations is their reliance on the visible light spectrum (RGB). This dependency assumes ideal lighting conditions, which are often unavailable in critical real-world scenarios. In applications such as search-and-rescue operations in smoke-filled buildings or autonomous navigation in total darkness, RGB cameras become ineffective.
Infrared (IR) cameras offer a robust alternative by detecting heat signatures rather than reflected visible light. Despite their utility, IR data presents unique challenges for neural reconstruction. IR images typically suffer from low contrast, sparse textures, and "thermal drift"—a phenomenon where the same temperature value is mapped to different pixel intensities across different frames. Standard NeRF optimization, which depends on pixel-level consistency across views, often fails to converge or produces significant "ghosting" artifacts when processing raw IR data. Thermal-NeRF addresses these challenges by introducing a framework specifically designed for the thermal spectrum.
Methodology
Thermal-NeRF adapts the standard NeRF pipeline through three primary innovations designed to handle the specific characteristics of infrared imagery.
Global Thermal Mapping
Standard image processing often involves frame-by-frame normalization, which destroys the absolute thermal consistency required for 3D reconstruction. Thermal-NeRF employs a global thermal mapping technique, applying min-max scaling across the entire image sequence. This ensures that a specific temperature value corresponds to a consistent 8-bit intensity across all viewpoints, maintaining the photometric consistency essential for NeRF optimization.
Structural Thermal Constraint ($L_{str}$)
Because IR images lack the sharp, high-frequency textures found in RGB data, traditional pixel-wise loss functions are often insufficient. The authors introduce a Heat-based Structural Similarity (HSSIM) index. This metric focuses on the structural similarities between rendered and ground-truth patches rather than simple luminance. To calculate this loss effectively during training, the model utilizes stochastic patch generation from minibatches of pixels, allowing the network to learn non-local structural information and recover geometry in low-contrast regions.
Joint Pose Refinement
Traditional Structure-from-Motion (SfM) tools, such as COLMAP, frequently struggle with the feature-sparse nature of IR images. To compensate for potential inaccuracies in initial camera poses, Thermal-NeRF jointly optimizes the camera poses and the neural radiance field. This refinement process ensures that the geometry and the camera trajectories are mutually consistent, leading to sharper reconstructions.
Experimental Setup and Datasets
As no standard benchmarks existed for IR-based NeRFs, the authors developed and open-sourced a new IR Dataset. The data was captured using an Optris PI 450i IR camera, with ground-truth camera poses recorded via a VICON Motion Capture system to ensure high-accuracy evaluation. The dataset covers three challenging environmental scenarios:
- Low-Light: Environments with zero visible light.
- Light-Changing: Scenarios with volatile illumination that would baffle RGB sensors.
- Smoky Conditions: Environments where visible light is obscured by particulates.
The framework was benchmarked against several state-of-the-art models, including the original NeRF, Mip-NeRF 360, and DVGO.
Results and Insights
Thermal-NeRF demonstrates a significant performance leap over existing methods in the infrared domain. Quantitatively, in low-light tests, Thermal-NeRF achieved a PSNR of 32.41, substantially higher than the original NeRF (27.57) and DVGO (19.18).
Qualitatively, the model produces sharp novel views with clear boundaries, avoiding the blurriness and artifacts seen in baseline models. Furthermore, when extracting 3D geometry via the Marching Cubes algorithm, Thermal-NeRF successfully reconstructed "hole-free" meshes of heat sources, such as kettles and cups. In contrast, Mip-NeRF 360 produced noisy and fragmented geometry under the same conditions.
A key insight from this research is the treatment of IR intensity as a physical thermal value rather than a simple grayscale pixel. By prioritizing the distribution and shape of heat (structure) over exact single-pixel intensity, the model becomes significantly more robust to sensor noise and the inherent limitations of the IR spectrum.
Limitations and Future Work
While Thermal-NeRF represents a breakthrough in thermal 3D reconstruction, it has limitations. The model currently excels at reconstructing localized heat sources but provides less detail in "cold" areas where the IR signal is weak. Additionally, the current framework is designed for static scenes; adapting the technology to dynamic, high-heat environments—such as active fire zones—remains a challenge for future research. The authors also suggest that integrating depth supervision from sensors like LiDAR could further enhance full-scene reconstruction accuracy.
Summary updated