Abstract
Traditional object tracking often fails in scenarios where targets and backgrounds share similar colors but different physical compositions. This paper introduces a paradigm shift toward material-based tracking using Hyperspectral Imaging (HSI). By leveraging the rich spectral fingerprints of materials, the authors propose a novel Material-based Hyperspectral Tracker (MHT) and a comprehensive benchmark dataset. The MHT integrates spectral-spatial gradients and material abundance maps into a correlation filter framework, demonstrating superior robustness in complex environments compared to state-of-the-art RGB and deep learning trackers.
The Challenge of Metamerism in Tracking
Standard object trackers rely on RGB color intensities, which are highly susceptible to failure in "metameric" scenarios. Metamerism occurs when a target and background appear visually identical in color but possess different material properties—for example, a green plastic toy against real grass. In such cases, traditional trackers often drift or lose the target entirely due to background clutter or rapid appearance changes.
Hyperspectral images provide a solution by capturing dozens or hundreds of narrow spectral bands, creating a unique "spectral fingerprint" for every material. However, the high dimensionality of HSI data and the historical lack of annotated video datasets have limited its application in real-time computer vision.
Methodology: The MHT Framework
The proposed Material-based Hyperspectral Tracker (MHT) addresses these challenges by extracting two primary types of features within a Background-Aware Correlation Filter (BACF) framework:
- Spectral-Spatial Histogram of Multidimensional Gradients (SSHMG): This 3D extension of the HOG descriptor captures local texture by calculating gradients across both spatial ($x, y$) and spectral ($k$) dimensions. This approach is significantly more stable under varying illumination than raw spectral values.
- Material Abundance Learning: Using semi-supervised hyperspectral unmixing, the tracker decomposes pixels into fractional abundances of constituent materials. This allows the system to distinguish the target's material composition from the background.
- Group-wise Reliability Learning: The tracker adaptively weights feature groups based on their recent performance, helping to suppress noise during rapid movement or occlusions.
The Hyperspectral Tracking Benchmark
To validate this approach, the authors contributed a new benchmark dataset consisting of 35 pairs of fully annotated HSI and RGB video sequences. Captured at 25 fps with 16 spectral bands (470nm to 620nm), the dataset includes 11 challenging attributes such as out-of-view movement, occlusion, and deformation. This represents one of the first large-scale datasets designed specifically for close-range hyperspectral tracking.
Key Results and Insights
Experimental results demonstrate that material properties are far more stable than color or shape for tracking purposes. The MHT achieved an Area Under Curve (AUC) of 61.2%, notably outperforming high-end deep trackers like ECO (58.6%) and C-COT (56.1%) on hyperspectral data.
Specifically, the MHT showed a 7.2% gain in AUC over the standard BACF baseline, proving the efficacy of material-based features. The tracker excelled in "Background Clutter" scenarios where visual similarity usually causes failure, ranking first across five of the eleven challenging attributes in the benchmark.
Conclusion and Future Directions
While the MHT provides a robust framework for material-based tracking, it still faces challenges under extreme illumination variations where initial material signatures may shift. Future research aims to optimize the computational overhead of hyperspectral unmixing and explore the development of "Material-based Convolutional Neural Networks" to learn even more discriminative spectral-spatial features directly from raw HSI data.
Summary updated