Coordinate-Aware Thermal Infrared Tracking Via Natural Language Modeling

Miao Yan, Ping Zhang, Haofei Zhang, Ruqian Hao, Juanxiu Liu, Xiaoyang Wang, Lin Liu

Abstract

Thermal infrared (TIR) object tracking is a critical component for all-weather surveillance and rescue operations, yet it faces significant challenges due to low signal-to-noise ratios and a lack of visual texture. This paper introduces NLMTrack, a novel framework that reimagines TIR tracking as a natural language modeling task. By employing an encoder-decoder Transformer architecture, NLMTrack "translates" visual features into discrete coordinate sequences. This approach, combined with a multi-level progressive fusion module, achieves state-of-the-art performance across major benchmarks while maintaining real-time processing speeds of 33 FPS.

Introduction

Unlike visible-spectrum imaging, thermal infrared (TIR) tracking relies on heat signatures, making it indispensable for night-time and adverse weather conditions. However, the inherent characteristics of TIR images—low resolution, lack of color, and poor contrast—render traditional tracking methods less effective. Current deep learning trackers often utilize complex regression heads to predict bounding boxes, frequently overlooking the inherent temporal and spatial correlations in video sequences.

NLMTrack addresses these limitations by framing tracking as a sequence-to-sequence generation task. By discretizing bounding box coordinates into a sequence of "words" or tokens, the model leverages the generative power of language models to maintain spatial consistency and robustly track targets in challenging environments.

Proposed Method: NLMTrack Architecture

The core innovation of NLMTrack lies in its unified approach to feature extraction, fusion, and coordinate prediction.

Unified Encoder and MLPFM

Traditional Siamese trackers extract features from the template and search region independently. In contrast, NLMTrack uses a single Unified Transformer Encoder to perform extraction and fusion concurrently, facilitating a deeper exchange of information. To compensate for the lack of detail in TIR images, the model incorporates a Multi-level Progressive Fusion Module (MLPFM). This module utilizes a feature pyramid structure with "UpFusion" and "DownFusion" components to integrate high-level semantic information with fine-grained spatial details across multiple scales.

The NLMTrack architecture, illustrating the unified encoder, multi-level progressive fusion module, and the causal decoder for coordinate sequence generation.

Causal Decoder and Coordinate Modeling

The tracking process is treated as a translation from visual features to a coordinate sequence $[x, y, w, h]$. A causal Transformer decoder takes the fused visual features and previously generated tokens to predict the next token autoregressively. Coordinates are discretized into 4,000 "bins," allowing the model to learn the underlying structure of object motion more effectively than traditional numerical regression.

Furthermore, NLMTrack introduces an elegant template update strategy. It uses the mean softmax score from the decoder as a built-in reliability metric. If the model's confidence in its generated tokens is low, it avoids updating the template with potentially noisy data, enhancing long-term stability.

Experimental Results and Analysis

NLMTrack was evaluated on three major TIR benchmarks: VOT-TIR2015, PTB-TIR, and the large-scale LSOTB-TIR dataset. The model consistently outperformed existing state-of-the-art (SOTA) trackers.

  • VOT-TIR2015: Achieved a top Expected Average Overlap (EAO) of 0.392.
  • PTB-TIR: Reached a Success Rate (Suc) of 0.693, a 5% improvement over previous Siamese-based models.
  • LSOTB-TIR: On this highly diverse dataset, it achieved a Suc of 0.716 and a Precision of 0.853.
Success plots across nine challenging attributes, showing NLMTrack (blue line) consistently outperforming other state-of-the-art trackers.

Beyond accuracy, NLMTrack maintains high efficiency. Running at 33 FPS on an NVIDIA RTX 3090, it meets the requirements for real-time deployment in practical applications like maritime rescue and night surveillance.

Conclusion

By unifying feature extraction and fusion within a natural language modeling paradigm, NLMTrack provides a robust and efficient solution for thermal infrared tracking. The transition from regression-based heads to sequence generation allows the model to better capture spatial-temporal correlations, setting a new standard for performance in the TIR domain. Future work may explore extending this framework to multi-task learning, such as simultaneous tracking and segmentation.

Summary updated

BibTeX
@article{yan2025coordinate,
  author = {Miao Yan and Ping Zhang and Haofei Zhang and Ruqian Hao and Juanxiu Liu and Xiaoyang Wang and Lin Liu},
  title = {{Coordinate-Aware Thermal Infrared Tracking Via Natural Language Modeling}},
  journal = {Expert Systems with Applications},
  year = {2025},
  doi = {10.1016/j.eswa.2024.126012},
}
Figure