TRNeRF: Restoring Blurry Rolling Shutter and Noisy Thermal Images with Neural Radiance Fields

Spencer Carmichael, Manohar Bhat, Mani Ramanagopal, Austin Buchan, Ram Vasudevan, Katherine A. SkinnerWACV 2025

TL;DR

This paper introduces TRNeRF, a novel method that uses Neural Radiance Fields (NeRFs) to restore thermal images degraded by motion blur, rolling shutter, and fixed pattern noise. By incorporating a microbolometer image formation model into the NeRF rendering pipeline, TRNeRF can generate sharp, clear thermal images even under challenging motion conditions, outperforming existing methods.

What Problem Does This Paper Address?

Thermal cameras are invaluable for applications like building inspections, search and rescue, and autonomous driving. However, affordable uncooled microbolometer thermal cameras suffer from significant image degradations: motion blur, rolling shutter distortions, and fixed pattern noise. These issues severely limit their usability, especially in dynamic scenarios with fast camera movement or low thermal contrast. While prior work has tackled these problems individually, TRNeRF aims to jointly address all three, a feat that has been largely unaddressed, particularly for real-world microbolometer data. Existing methods often rely on synthetic data or are limited to specific camera types and scenarios, leaving a gap in robust thermal image restoration.

Methods

TRNeRF builds upon the foundation of Neural Radiance Fields (NeRFs) [28], a powerful technique for representing 3D scenes implicitly. The core innovation lies in modifying the NeRF rendering pipeline to account for the specific image formation characteristics of microbolometers.

  • Microbolometer Image Formation Model: The paper explicitly models the thermal time constant that causes motion blur in microbolometers, which differs from the finite exposure model in visible light cameras. It also incorporates the rolling shutter readout mechanism, where pixels are captured sequentially.
  • Joint Degradation Handling: During training, TRNeRF models these degradations (motion blur, rolling shutter, and fixed pattern noise) within the rendering process. This allows the NeRF to learn a representation of the scene that, when rendered, approximates the degraded input images.
  • Fixed Pattern Noise (FPN) Correction: A learnable per-pixel offset is introduced to model and correct for slowly varying FPN.
  • Inference: At inference time, these degradation models are removed, enabling the NeRF to render sharp, global shutter, and clear thermal images.
  • NeRF Architecture: The authors leverage Instant-NGP [29] for its efficient training and rendering capabilities.

Datasets

The paper introduces a new real-world dataset collected using a custom rig equipped with microbolometer thermal cameras, monochrome cameras, and an IMU. This dataset is designed to support quantitative evaluation of thermal image restoration methods. It includes sequences captured under varying conditions:

  • Outdoor and indoor scenes.
  • Slow, medium, and fast camera motion.
  • High and low thermal contrast environments.

The dataset is made publicly available.

Key Insights and Ideas

  • Microbolometer Physics Matters: The paper highlights that simply applying visible-light image restoration techniques to thermal data is insufficient due to the unique physics of microbolometers (e.g., thermal time constant for blur).
  • Joint Restoration is Crucial: Addressing motion blur, rolling shutter, and FPN simultaneously is key to achieving significant improvements, especially in challenging, high-motion scenarios where single-task methods fail.
  • Novel Evaluation Strategies: The introduction of evaluation metrics that do not require ground truth restored images is a significant contribution, addressing a common bottleneck in thermal image research. The "detection percentage" using AprilTags and "pseudo ground truth" generation are clever approaches.
  • FPN is a Major Hurdle: The ablation studies clearly demonstrate the importance of correcting fixed pattern noise, particularly in indoor scenes with low thermal contrast.

Key Results

  • TRNeRF successfully restores sharp, global shutter, and clear thermal images, even under extremely aggressive camera motion where existing methods fail.
  • Qualitative results (Figure 5) show a dramatic improvement in image quality for TRNeRF compared to baselines and raw images, especially in medium and fast sequences.
  • Quantitative results (Table 1) show TRNeRF achieving significantly higher detection percentages on the AprilTag board in medium and fast sequences compared to baselines like USB-NeRF and GS on the Move. For instance, in the fast outdoor sequence, TRNeRF achieves 58.6% detection, while USB-NeRF and GS on the Move achieve 0.0%.
  • The LPIPS scores (lower is better) also indicate TRNeRF's superior performance, particularly in the fast sequences, suggesting better perceptual quality.

Limitations and Future Work

  • Pose Estimation: While the paper uses COLMAP for pose estimation, it notes that COLMAP struggles with the motion blur and rolling shutter distortions in thermal images. The authors rely on transforming poses from monochrome cameras, which might introduce some inaccuracies.
  • Pseudo Ground Truth Limitations: The pseudo ground truth generation method relies on the assumption that a NeRF can be accurately trained on slow sequences. This might limit its applicability if even slow sequences are too degraded.
  • Future Work: The authors mention providing IMU data to support future work, suggesting potential for incorporating IMU data for more robust pose estimation or scene understanding. They also suggest that their method is conceptually agnostic to the underlying NeRF architecture, implying potential for integration with other advanced NeRF variants.

Summary updated

BibTeX
@inproceedings{carmichael2025trnerf,
  author = {Spencer Carmichael and Manohar Bhat and Mani Ramanagopal and Austin Buchan and Ram Vasudevan and Katherine A. Skinner},
  title = {{TRNeRF: Restoring Blurry Rolling Shutter and Noisy Thermal Images with Neural Radiance Fields}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure