Coordinate-aware thermal infrared tracking via natural language modeling

Miao Yan, Ping Zhang, Haofei Zhang, Ruqian Hao, Juanxiu Liu, Xiaoyang Wang, Lin Liu

TL;DR

NLMTrack reimagines thermal infrared (TIR) object tracking as a natural language modeling task, using an encoder-decoder Transformer to "translate" visual features into coordinate sequences. By unifying feature extraction and fusion with a novel multi-level progressive module, it achieves state-of-the-art accuracy on major benchmarks while maintaining real-time performance at 33 FPS.

What Problem Does This Paper Address?

Thermal infrared (TIR) tracking is essential for "all-weather" applications like maritime rescue and night surveillance because it relies on heat signatures rather than visible light. However, TIR images are notoriously difficult to work with: they lack color and texture, suffer from low signal-to-noise ratios, and often have very low contrast.

Current deep learning approaches typically rely on Siamese networks or standard Transformers that use complex "heads" to regress bounding boxes. These methods often treat feature extraction and fusion as separate steps and frequently overlook the temporal and coordinate correlations inherent in a video sequence. The authors argue that by treating the bounding box coordinates as a sequence of "words," we can leverage the powerful generative capabilities of language models to track targets more robustly.

Methods

The core of NLMTrack is the Natural Language Modeling paradigm. Instead of predicting a bounding box via regression, the model generates a sequence of tokens $[x, y, w, h]$ step-by-step.

Key technical innovations include:

  • Unified Encoder: Unlike traditional Siamese trackers that extract features from the template and search region separately before fusing them, NLMTrack uses a single Transformer encoder to perform extraction and fusion concurrently. This allows for better information exchange between the target and the search area.
  • Multi-level Progressive Fusion Module (MLPFM): To handle the lack of detail in TIR images, this module uses a feature pyramid approach. It employs "UpFusion" and "DownFusion" modules to integrate high-level semantic information with fine-grained spatial details across multiple scales.
  • Causal Decoder: A causal Transformer decoder takes the visual features and previously generated coordinate tokens to predict the next token in the sequence. This "autoregressive" approach helps the model maintain spatial consistency.
  • Adaptive Loss & Template Update: The authors combine standard cross-entropy loss with SIOU (Scylla-IoU) loss to refine spatial accuracy. Furthermore, they use the mean softmax score from the decoder as a built-in "reliability metric" to decide when to update the target template, avoiding the need for a separate, complex scoring branch.

Datasets

The model was trained on LSOTB-TIR and the visible-spectrum dataset GOT-10k. Evaluation was performed on three major TIR benchmarks:

  • VOT-TIR2015: A standard benchmark with 20 sequences.
  • PTB-TIR: A pedestrian-focused benchmark with 60 sequences.
  • LSOTB-TIR: A large-scale, high-diversity dataset containing 1,280 sequences, which is currently the most challenging benchmark in the field.

Key Insights and Ideas

  • Tracking as Translation: The most striking "aha moment" is the realization that tracking can be framed as a sequence-to-sequence task. The search region is the "source language," and the bounding box coordinates are the "target language."
  • Coordinate Awareness: By discretizing coordinates into 4,000 "bins" (tokens), the model learns the underlying structure of how objects move and change size more effectively than simple numerical regression.
  • Softmax as Confidence: Using the internal probability of the language model (the softmax score) to assess tracking reliability is an elegant way to handle template updates. If the model is "unsure" about its generated tokens, it won't update the template with potentially noisy data.
  • Simplicity Wins: By abandoning complex regression heads and separate fusion layers, the architecture becomes more streamlined and easier to train, yet more powerful.

Key Results

NLMTrack consistently outperformed existing state-of-the-art (SOTA) trackers across all tested datasets:

  • VOT-TIR2015: Achieved a top EAO (Expected Average Overlap) of 0.392, significantly outperforming recent models like SiamMSS and DFG.
  • PTB-TIR: Reached a Success Rate (Suc) of 0.693, a 5% improvement over the previous best-performing Siamese-based models.
  • LSOTB-TIR: On the largest benchmark, it achieved a Suc of 0.716 and a Precision of 0.853, proving its robustness in complex, real-world scenarios.
  • Efficiency: Despite the complexity of the Transformer architecture, the model runs at 33 FPS on an NVIDIA RTX 3090, making it suitable for real-time deployment.

Limitations and Future Work

While NLMTrack shows impressive results, the authors note that the "natural language interface" for vision tasks is still in its early stages for TIR tracking.

  • Multi-Task Potential: The authors suggest that this framework could be extended to multi-task learning, perhaps combining tracking with segmentation or action recognition within the same sequence-generation pipeline.
  • Vocabulary Sensitivity: The performance relies on the quantization of coordinates (the number of "bins"). While 4,000 bins worked well here, finding the optimal balance between discretization and precision remains an area for further exploration.
  • Temporal Window: Currently, the model focuses heavily on intra-frame prediction; incorporating longer-term temporal memory (beyond the simple template update) could further improve performance during long-term occlusions.

Summary updated

BibTeX
@article{yan2025coordinate,
  author = {Miao Yan and Ping Zhang and Haofei Zhang and Ruqian Hao and Juanxiu Liu and Xiaoyang Wang and Lin Liu},
  title = {{Coordinate-aware thermal infrared tracking via natural language modeling}},
  journal = {Expert Systems with Applications},
  year = {2025},
  doi = {10.1016/j.eswa.2024.126012},
}
Figure