Global Motion-Aware Robust Visual Object Tracking for Electro Optical Targeting Systems

Byeong Hak Kim, Alan Lukezic, Jong Hyuk Lee, Ho Min Jung, Min Young KimSensors 2020 · OpenReview

TL;DR

This paper introduces a novel "global motion-aware" tracking method designed to significantly improve the robustness of visual object trackers in electro-optical targeting systems (EOTS). By detecting and compensating for fast camera movements using image entropy and background tracking, the method enhances the performance of existing trackers with minimal computational overhead, making it suitable for real-time applications.

What Problem Does This Paper Address?

Visual object tracking is crucial for systems like electro-optical targeting systems (EOTS) used in defense and security. However, these systems often operate in challenging environments with frequent and abrupt camera movements (e.g., from pan-tilt motors or vehicle dynamics). These rapid motions can cause significant image blur, especially in thermal infrared (TIR) imaging due to longer integration times. Existing advanced trackers, while good in many scenarios, struggle to maintain track when faced with these fast camera motions, leading to target loss. This paper aims to solve this by explicitly detecting and compensating for global camera motion.

Methods

The proposed method, termed Global Motion-Aware Robust Visual Object Tracking (GESBT), consists of two main modules:

  1. Motion Detection Module (Gradient of Entropy Sensor - GES): This module detects when fast camera motion is occurring. It leverages the principle that image blur, often caused by fast camera movement, leads to a decrease in image entropy. The GES module specifically looks at the temporal derivative of image entropy for both thermal infrared (IR) and RGB images (ΔEIR and ΔERGB), along with the tracker's correlation response (R(t)). A combination of these signals, with specific thresholds tuned for IR and RGB sensors, triggers the motion detection. This approach is designed to be less sensitive to background variations than using entropy alone.
  2. Background Tracking Module (BT): Once motion is detected, this module estimates the global camera movement. It does this by tracking features between consecutive frames. To ensure real-time performance, the authors employ a modified Shi-Tomasi corner detection method, which is known for its speed and effectiveness, even on blurry images. An adaptive quality threshold for feature detection is introduced to better handle the often poor texture in IR images. The estimated motion vector is then used to adjust the search region for the primary object tracker, preventing the target from being lost due to large camera-induced displacements.

The GESBT framework is designed to be easily integrated into existing correlation filter-based trackers.

Datasets

The paper evaluates its method on several datasets:

  • VOT2019 RGB-TIR dataset: This dataset is highlighted as challenging and relevant to real EOTS applications, containing sequences with significant camera motion.
  • PTB-TIR and RGB-T234 benchmarks: These datasets are used for further validation, with RGB-T234 specifically noted for its 89 sequences with fast camera motion.
  • Real EOTS datasets: The authors also present results from experiments conducted on a hard-mounted EOTS product in a controlled environment, simulating various camera movements and target motions.

Key Insights and Ideas

  • Entropy as a Motion Indicator: The core idea of using image entropy to detect blur caused by camera motion is clever. The paper refines this by focusing on the change in entropy (temporal derivative) and combining it with other cues (RGB entropy, correlation response) to create a more robust motion trigger.
  • Adaptive Feature Detection for IR: The challenge of poor texture in IR images is addressed by proposing an adaptive quality threshold for feature detection, which is crucial for reliable background tracking in these conditions.
  • Minimal Computational Overhead: A significant insight is that robustly handling fast camera motion doesn't require a massive increase in computational cost. The proposed GESBT method adds only a small overhead, making it practical for real-time EOTS.
  • Modularity is Key: The GESBT framework is presented as a plug-in module, demonstrating its versatility and ease of integration with various existing trackers.

Key Results

  • Improved Robustness: The GESBT method significantly improved the robustness of four baseline trackers (MOSSE_CA, STAPLE, STAPLE_CA, DFPReco) across various datasets. For example, it reduced the failure rate of STAPLE_CA by 4.03% and DFPReco by 2.87% on the VOT2019 dataset.
  • Performance Under Camera Motion: When specifically evaluated on sequences with camera motion, the proposed method showed even greater improvements in robustness.
  • Quantitative Metrics: The paper reports improvements in metrics like Expected Average Overlap (EAO), precision, and success rate. For instance, STAPLE+Proposed achieved an EAO of 0.28, compared to 0.28 for STAPLE alone, but with significantly fewer failures.
  • Qualitative Success: Visual examples demonstrate that baseline trackers often lose the target after camera motion, while trackers augmented with GESBT successfully maintain tracking.
  • Speed: While the method adds some computational cost, it's noted as minimal. For example, when applied to STAPLE, it reduced speed by only 8%, compared to a 29% reduction when a context-aware (CA) method was applied.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the method improves robustness, it doesn't eliminate all tracking failures, especially in extremely challenging scenarios. The effectiveness of the GES module relies on the specific thresholds (AIR, ARGB) which are sensor-specific and require tuning.
  • Future Work: The authors suggest incorporating deep Convolutional Neural Network (CNN) features for more sophisticated global motion estimation. They also propose formulating the problem as an end-to-end training task, which could potentially lead to even more integrated and optimized solutions.

Summary updated

Abstract

Although recently developed trackers have shown excellent performance even when tracking fast moving and shape changing objects with variable scale and orientation, the trackers for the electro-optical targeting systems (EOTS) still suffer from abrupt scene changes due to frequent and fast camera motions by pan-tilt motor control or dynamic distortions in field environments. Conventional context aware (CA) and deep learning based trackers have been studied to tackle these problems, but they have the drawbacks of not fully overcoming the problems and dealing with their computational burden. In this paper, a global motion aware method is proposed to address the fast camera motion issue. The proposed method consists of two modules: (i) a motion detection module, which is based on the change in image entropy value, and (ii) a background tracking module, used to track a set of features in consecutive images to find correspondences between them and estimate global camera movement. A series of experiments is conducted on thermal infrared images, and the results show that the proposed method can significantly improve the robustness of all trackers with a minimal computational overhead. We show that the proposed method can be easily integrated into any visual tracking framework and can be applied to improve the performance of EOTS applications.
BibTeX
@inproceedings{kim2020global,
  author = {Byeong Hak Kim and Alan Lukezic and Jong Hyuk Lee and Ho Min Jung and Min Young Kim},
  title = {{Global Motion-Aware Robust Visual Object Tracking for Electro Optical Targeting Systems}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=aC5bZXZxoE},
}
Figure