Material Based Object Tracking in Hyperspectral Videos: Benchmark and Algorithms

Fengchao Xiong, Jun Zhou, Yuntao Qian

TL;DR

This paper introduces a novel approach to object tracking by leveraging hyperspectral imaging (HSI) to identify targets based on their unique material signatures rather than just RGB color. The authors provide the first large-scale, fully-annotated hyperspectral video benchmark and a new tracking algorithm (MHT) that outperforms state-of-the-art deep learning and hand-crafted trackers in challenging scenarios like background clutter and object deformation.

What Problem Does This Paper Address?

Standard computer vision relies on RGB images, which collapse the complex visible spectrum into just three channels. This leads to a major limitation in object tracking: metamerism. Two objects might look identical in color (e.g., a real lemon and a plastic lemon) but consist of entirely different materials. When a target moves in front of a background with a similar color, or undergoes rapid shape changes, traditional trackers often "drift" and lose the target.

While hyperspectral imaging—which captures dozens or hundreds of narrow spectral bands—offers a solution by revealing the "material fingerprint" of objects, the field has lacked two things:

  1. Data: There were no large-scale, high-frame-rate hyperspectral video datasets for close-range tracking.
  2. Algorithms: Existing HSI trackers often ignored the spatial-spectral correlation or relied on converting HSI data back into "false-color" RGB, throwing away the very information that makes HSI valuable.

Methods

The authors propose the Material-based Hyperspectral Tracker (MHT), which integrates two novel feature types into a Background-Aware Correlation Filter (BACF) framework:

  • Spectral-Spatial Histogram of Multidimensional Gradients (SSHMG): Instead of looking at 2D gradients (like HOG), SSHMG treats the HSI data as a 3D tensor. It calculates gradients across the $x$, $y$, and spectral dimensions, capturing the local texture and spectral transitions simultaneously.
  • Material Abundances: Using a process called hyperspectral unmixing, the algorithm decomposes each pixel into a "recipe" of constituent materials (endmembers). This allows the tracker to understand the underlying material distribution, which remains stable even if the object rotates or deforms.

To combine these, the authors developed an online learning approach that adaptively weights these features based on their reliability (measured by overlap, distance, and trajectory smoothness). If one feature becomes noisy (e.g., due to a sudden shadow), the tracker automatically relies more on the other.

Datasets

The authors introduced a new benchmark dataset to the community:

  • 35 Hyperspectral Videos: Captured at 25 frames per second using a snapshot mosaic camera.
  • 16 Spectral Bands: Ranging from 470nm to 620nm.
  • Paired RGB Data: For every HSI sequence, they provided a corresponding color video of the same scene for fair comparison.
  • Annotations: Fully annotated with bounding boxes and 11 challenging attributes (e.g., occlusion, fast motion, illumination variation).

Key Insights and Ideas

  • Material is more robust than color: The "aha moment" is shown in Figure 1, where two toy minions look identical in color, but their spectral responses are vastly different. This difference is the key to distinguishing a target from a "camouflaged" background.
  • 3D is better than 2D: By treating HSI as a 3D cube rather than a stack of 2D images, the SSHMG feature captures the physical structure of the material more effectively than standard computer vision features like HOG.
  • Unmixing as a "Bag-of-Materials": Using fractional abundances acts similarly to a "bag-of-words" model but for physical matter. It provides a high-level semantic understanding of what the object is made of, which is invariant to many geometric changes.

Key Results

  • Superior Accuracy: The proposed MHT achieved an Area Under Curve (AUC) of 61.2%, outperforming top-tier RGB trackers like C-COT (61.7% on color, but lower on HSI-derived data) and ECO.
  • Winning in the Clutter: MHT ranked first on 5 out of 11 challenging attributes, specifically excelling in In-Plane Rotation (IPR), Out-of-Plane Rotation (OPR), Deformation (DEF), Background Clutter (BC), and Out-of-View (OV).
  • Comparison to HSI Trackers: MHT significantly outperformed previous HSI-specific trackers like DeepHKCF (34.2% AUC) and CNHT (13.3% AUC), proving that their specific feature extraction methods are much more effective at utilizing spectral data.

Limitations and Future Work

The primary limitation noted is Illumination Variation (IV). Because the material "endmembers" are extracted at the start of the sequence and held relatively fixed, a dramatic change in lighting can alter the perceived spectral signature, causing the unmixing process to fail.

The authors suggest that future work should involve developing material-based convolutional neural networks (CNNs) to learn even deeper spectral-spatial representations, potentially moving away from hand-crafted features like SSHMG toward end-to-end deep hyperspectral tracking.

Summary updated

Abstract

Traditional color images only depict color intensities in red, green and blue channels, often making object trackers fail in challenging scenarios, e.g., background clutter and rapid changes of target appearance. Alternatively, material information of targets contained in a large amount of bands of hyperspectral images (HSI) is more robust to these difficult conditions. In this paper, we conduct a comprehensive study on how material information can be utilized to boost object tracking from three aspects: benchmark dataset, material feature representation and material based tracking. In terms of benchmark, we construct a dataset of fully-annotated videos, which contain both hyperspectral and color sequences of the same scene. Material information is represented by spectral-spatial histogram of multidimensional gradient, which describes the 3D local spectral-spatial structure in an HSI, and fractional abundances of constituted material components which encode the underlying material distribution. These two types of features are embedded into correlation filters, yielding material based tracking. Experimental results on the collected benchmark dataset show the potentials and advantages of material based object tracking.
BibTeX
@article{xiong2020material,
  author = {Fengchao Xiong and Jun Zhou and Yuntao Qian},
  title = {{Material Based Object Tracking in Hyperspectral Videos: Benchmark and Algorithms}},
  journal = {IEEE Transactions on Image Processing},
  year = {2020},
  doi = {10.1109/tip.2020.2965302},
}

Loading related content...

Figure