TP-TIO: A Robust Thermal-Inertial Odometry with Deep ThermalPoint

Shibo Zhao, Peng Wang, Hengrui Zhang, Zheng Fang, Sebastian A. Scherer

Abstract

Reliable state estimation in visually degraded environments—such as smoke-filled corridors or lightless mines—remains a significant hurdle for autonomous systems. Traditional Visual Odometry (VO) and LiDAR-based methods often fail due to occlusion or lack of features. We present TP-TIO, a robust thermal-inertial odometry framework that utilizes a lightweight deep-learning feature detector, ThermalPoint, and a novel radiometric tracking method. By processing raw 16-bit thermal data and employing a noise-aware training strategy, TP-TIO achieves high-precision motion estimation that outperforms state-of-the-art laser and visual-inertial systems in challenging conditions.

Introduction

Standard navigation sensors like visible-light cameras and LiDAR face severe limitations in "denied" environments. Smoke, dust, and total darkness render RGB cameras useless, while LiDAR beams often reflect off airborne particles, leading to catastrophic drift. Thermal cameras offer a solution by sensing long-wave infrared radiation, but they introduce new challenges: low contrast, high fixed-pattern noise (FPN), and sudden photometric shifts during non-uniformity correction (NUC) cycles. TP-TIO addresses these issues by combining deep-learning-based feature detection with a tightly coupled thermal-inertial backend.

Methodology

The TP-TIO framework is built upon three core pillars: a specialized feature detector, radiometric tracking, and a robust optimization backend.

ThermalPoint Network

At the heart of the system is ThermalPoint, a lightweight convolutional neural network (CNN) based on the MobileNet architecture, optimized for real-time performance on embedded hardware. Unlike general-purpose detectors, ThermalPoint is trained using a noise-augmentation strategy that includes synthetic Fixed-Pattern Noise (FPN). This allows the network to become "blind" to common thermal sensor artifacts like vertical or horizontal stripes. The network employs an independent logistic loss function to maximize feature repeatability across frames.

Radiometric Feature Tracking

To preserve the maximum amount of information, TP-TIO tracks features using raw 16-bit radiometric values rather than compressed 8-bit images. We utilize an IMU-aided Kanade-Lucas-Tomasi (KLT) tracker that is invariant to intensity scaling. This approach ensures stability even during sudden brightness jumps caused by temperature shifts or sensor recalibration.

Tightly Coupled Backend

The system employs a sliding-window optimization that jointly minimizes visual reprojection errors and IMU preintegration errors. This tight coupling ensures that the IMU maintains state estimation during brief thermal sensor interruptions, such as NUC cycles.

The TP-TIO system architecture, featuring the ThermalPoint network for feature detection and the tightly coupled thermal-inertial optimization pipeline.

Experimental Results

The system was evaluated across synthetic datasets, public underground mine data, and custom indoor smoke environments.

Feature Repeatability

ThermalPoint demonstrated superior performance compared to both traditional and deep-learning baselines. It achieved a repeatability score of 0.7674 on thermal data, significantly higher than SuperPoint (0.7166) and SIFT (0.4976).

Performance in Degraded Environments

In dense smoke environments, TP-TIO maintained a stable trajectory where LiDAR-based methods (LOAM) failed. LiDAR beams reflected off smoke particles, causing massive point cloud misalignment and z-axis drift. In contrast, TP-TIO's thermal-based tracking remained unaffected by the smoke.

Comparison of TP-TIO and LOAM in smoke-filled environments. (a-b) show the visual degradation, (c) shows ThermalPoint tracking, and (d-e) illustrate the significant drift and misalignment in LiDAR-based mapping compared to the stable TP-TIO trajectory.

Accuracy and Real-time Viability

In indoor testing, TP-TIO achieved an Absolute Trajectory Error (ATE) of 0.359m, outperforming VINS-Mono (2.406m) and OKVIS (3.199m) when applied to thermal imagery. The entire pipeline runs in real-time on a Jetson AGX Xavier, confirming its suitability for deployment on robotic platforms.

Conclusion and Future Work

TP-TIO provides a robust solution for odometry in environments where traditional sensors fail. By integrating a noise-aware deep detector with raw radiometric tracking, the system achieves high accuracy and reliability. Future work will focus on developing a loop closure module to enable global consistency and large-scale mapping capabilities.

Summary updated

Abstract

To achieve robust motion estimation in visually degraded environments, thermal odometry has been an attraction in the robotics community. However, most thermal odometry methods are purely based on classical feature extractors, which is difficult to establish robust correspondences in successive frames due to sudden photometric changes and large thermal noise. To solve this problem, we propose ThermalPoint, a lightweight feature detection network specifically tailored for producing keypoints on thermal images, providing notable anti-noise improvements compared with other state-of-the-art methods. After that, we combine ThermalPoint with a novel radiometric feature tracking method, which directly makes use of full radiometric data and establishes reliable correspondences between sequential frames. Finally, taking advantage of an optimization-based visual-inertial framework, a deep feature-based thermal-inertial odometry (TP-TIO) framework is proposed and evaluated thoroughly in various visually degraded environments. Experiments show that our method outperforms state-of-the-art visual and laser odometry methods in smoke-filled environments and achieves competitive accuracy in normal environments.
BibTeX
@inproceedings{zhao2020tp,
  author = {Shibo Zhao and Peng Wang and Hengrui Zhang and Zheng Fang and Sebastian A. Scherer},
  title = {{TP-TIO: A Robust Thermal-Inertial Odometry with Deep ThermalPoint}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=95IMYjCHWQg},
}
Figure