TL;DR
This paper introduces a novel "global motion-aware" tracking method designed to significantly improve the robustness of visual object trackers in electro-optical targeting systems (EOTS). By detecting and compensating for fast camera movements using image entropy and background tracking, the method enhances the performance of existing trackers with minimal computational overhead, making it suitable for real-time applications.
What Problem Does This Paper Address?
Visual object tracking is crucial for systems like electro-optical targeting systems (EOTS) used in defense and security. However, these systems often operate in challenging environments with frequent and abrupt camera movements (e.g., from pan-tilt motors or vehicle dynamics). These rapid motions can cause significant image blur, especially in thermal infrared (TIR) imaging due to longer integration times. Existing advanced trackers, while good in many scenarios, struggle to maintain track when faced with these fast camera motions, leading to target loss. This paper aims to solve this by explicitly detecting and compensating for global camera motion.
Methods
The proposed method, termed Global Motion-Aware Robust Visual Object Tracking (GESBT), consists of two main modules:
- Motion Detection Module (Gradient of Entropy Sensor - GES): This module detects when fast camera motion is occurring. It leverages the principle that image blur, often caused by fast camera movement, leads to a decrease in image entropy. The GES module specifically looks at the temporal derivative of image entropy for both thermal infrared (IR) and RGB images (ΔEIR and ΔERGB), along with the tracker's correlation response (R(t)). A combination of these signals, with specific thresholds tuned for IR and RGB sensors, triggers the motion detection. This approach is designed to be less sensitive to background variations than using entropy alone.
- Background Tracking Module (BT): Once motion is detected, this module estimates the global camera movement. It does this by tracking features between consecutive frames. To ensure real-time performance, the authors employ a modified Shi-Tomasi corner detection method, which is known for its speed and effectiveness, even on blurry images. An adaptive quality threshold for feature detection is introduced to better handle the often poor texture in IR images. The estimated motion vector is then used to adjust the search region for the primary object tracker, preventing the target from being lost due to large camera-induced displacements.
The GESBT framework is designed to be easily integrated into existing correlation filter-based trackers.
Datasets
The paper evaluates its method on several datasets:
- VOT2019 RGB-TIR dataset: This dataset is highlighted as challenging and relevant to real EOTS applications, containing sequences with significant camera motion.
- PTB-TIR and RGB-T234 benchmarks: These datasets are used for further validation, with RGB-T234 specifically noted for its 89 sequences with fast camera motion.
- Real EOTS datasets: The authors also present results from experiments conducted on a hard-mounted EOTS product in a controlled environment, simulating various camera movements and target motions.
Key Insights and Ideas
- Entropy as a Motion Indicator: The core idea of using image entropy to detect blur caused by camera motion is clever. The paper refines this by focusing on the change in entropy (temporal derivative) and combining it with other cues (RGB entropy, correlation response) to create a more robust motion trigger.
- Adaptive Feature Detection for IR: The challenge of poor texture in IR images is addressed by proposing an adaptive quality threshold for feature detection, which is crucial for reliable background tracking in these conditions.
- Minimal Computational Overhead: A significant insight is that robustly handling fast camera motion doesn't require a massive increase in computational cost. The proposed GESBT method adds only a small overhead, making it practical for real-time EOTS.
- Modularity is Key: The GESBT framework is presented as a plug-in module, demonstrating its versatility and ease of integration with various existing trackers.
Key Results
- Improved Robustness: The GESBT method significantly improved the robustness of four baseline trackers (MOSSE_CA, STAPLE, STAPLE_CA, DFPReco) across various datasets. For example, it reduced the failure rate of STAPLE_CA by 4.03% and DFPReco by 2.87% on the VOT2019 dataset.
- Performance Under Camera Motion: When specifically evaluated on sequences with camera motion, the proposed method showed even greater improvements in robustness.
- Quantitative Metrics: The paper reports improvements in metrics like Expected Average Overlap (EAO), precision, and success rate. For instance, STAPLE+Proposed achieved an EAO of 0.28, compared to 0.28 for STAPLE alone, but with significantly fewer failures.
- Qualitative Success: Visual examples demonstrate that baseline trackers often lose the target after camera motion, while trackers augmented with GESBT successfully maintain tracking.
- Speed: While the method adds some computational cost, it's noted as minimal. For example, when applied to STAPLE, it reduced speed by only 8%, compared to a 29% reduction when a context-aware (CA) method was applied.
Limitations and Future Work
- Limitations: The paper acknowledges that while the method improves robustness, it doesn't eliminate all tracking failures, especially in extremely challenging scenarios. The effectiveness of the GES module relies on the specific thresholds (AIR, ARGB) which are sensor-specific and require tuning.
- Future Work: The authors suggest incorporating deep Convolutional Neural Network (CNN) features for more sophisticated global motion estimation. They also propose formulating the problem as an end-to-end training task, which could potentially lead to even more integrated and optimized solutions.
Summary updated