Exploring Local Context for Multi-target Tracking in Wide Area Aerial Surveillance

Bor-Jeng Chen, Gérard G. MedioniWACV 2017 · DOI

TL;DR

This paper tackles the challenging problem of multi-target tracking in Wide Area Motion Imagery (WAMI), where targets are small, grayscale, and often suffer from long-term occlusions. They introduce a novel approach that combines a Detection-Based Tracker (DBT) with a Local Context Tracker (LCT), significantly improving tracking robustness by leveraging spatial relationships between targets and handling merged detections more effectively.

What Problem Does This Paper Address?

Tracking multiple vehicles in Wide Area Motion Imagery (WAMI) is crucial for understanding events in large surveillance areas. WAMI data presents unique challenges: it's captured by high-altitude drones, resulting in large image formats, low frame rates (1-2 Hz), grayscale imagery, and small target sizes (often less than 20 pixels). Traditional tracking methods, like those relying on motion detection and frame differencing, struggle with slow or stopped vehicles, leading to long-term missing detections and track discontinuities. Appearance-based methods are also problematic due to the small target size and grayscale nature, making targets visually similar to their background and each other. Existing approaches often fail to handle split-and-merge detections, where multiple targets are detected as a single blob.

Methods

The paper proposes a unified framework that couples two main components:

  • Detection-Based Tracker (DBT): This component builds short tracks (tracklets) by associating motion detections within a sliding temporal window (8 frames). A key innovation here is the explicit handling of "merged detections." When a detection's size significantly increases compared to its parent detection, it's flagged as a "potential merged detection." The system then generates additional hypotheses from this region, improving the chances of finding a more accurate tracklet and avoiding suboptimal choices based on merged detections.
  • Local Context Tracker (LCT): This component addresses the limitations of appearance-based tracking and motion detection. Instead of relying solely on appearance, LCT explores the spatial relationships between a target and its "motion neighbors" (targets with similar motion directions). This context helps to avoid drifting to nearby distractors. Hypotheses for LCT are generated efficiently using both dense optical flow (for fast-moving targets) and sparse optical flow (for slow or stopped targets). The tracking problem is then formulated as a graph optimization problem, where the objective is to maximize a score that considers both individual target appearance/motion and the contextual relationships with neighbors.

The two trackers are integrated through a Track Association module. Tracklets from DBT and results from LCT are combined to update a "track pool," which stores all active tracks and is used to initialize LCT in subsequent frames.

Datasets

The paper evaluates its method on two WAMI sequences:

  • WPFAB dataset [11]: This dataset is described as a public WAMI dataset recorded at approximately 1 Hz, covering a 429m x 429m area with 1408x1408 pixel images and a ground sampling distance of 0.3 meters/pixel. It contains 1025 frames and 410 tracks, with scenarios including stop-then-go vehicles and merged detections.
  • Rochester dataset: This sequence is captured from Rochester, NY, USA, at 2 Hz with a ground sampling distance of 0.38 meters/pixel. It consists of 96 frames and 44 tracks, featuring urban scenarios with vehicles stopping at intersections for extended periods.

Key Insights and Ideas

  • Context is King (for tracking): The paper strongly advocates for using local context (relationships between nearby moving targets) as a robust cue for tracking, especially when appearance is unreliable. This is a departure from methods that focus solely on individual target appearance or motion.
  • Handling Merged Detections is Crucial: The DBT's ability to generate additional hypotheses from merged detections is a significant contribution, directly addressing a common failure mode in WAMI tracking.
  • Dual-Flow Hypothesis Generation: Efficiently generating hypotheses using both dense and sparse optical flow allows the LCT to handle a wide range of target speeds, from fast to stationary.
  • Relaxing Appearance Dependency: By introducing LCT, the framework reduces its reliance on appearance models, which are inherently weak in WAMI, making the system more robust to illumination changes and small target sizes.

Key Results

The proposed DBT+LCT method demonstrates significant improvements over state-of-the-art methods on both datasets.

  • WPFAB 2009 Sequence: The paper reports the best performance across several metrics, including Recall, Precision, MODA, S/T (swaps), B/T (breaks), and MOTA. Specifically, the DBT+LCT method achieved a Recall of 0.606 and MOTA of 0.599, outperforming other methods. The introduction of LCT significantly reduced the number of swaps (S/T) and breaks (B/T), indicating more consistent and accurate tracking.
  • Rochester Sequence: Again, the DBT+LCT method achieved the best results, with a Recall of 0.761 and MOTA of 0.761. Notably, the method produced zero swaps (S/T) on this sequence. The paper highlights that LCT recovers missing detections, increasing recall by up to 19% in move-then-stop scenarios where motion-only detectors fail.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that the DBT+LCT approach has a higher computational cost compared to simpler DBT methods, primarily due to the dense flow computation. While it's more efficient than some hybrid approaches, further optimization is desired.
  • Future Work: The authors plan to reduce computation time by leveraging parallel programming, as both DBT and LCT can be processed in parallel for each target. They also aim to infer track associations during long-term occlusions, a common challenge in urban surveillance scenarios.

Summary updated

BibTeX
@inproceedings{chen2017exploring,
  author = {Bor-Jeng Chen and Gérard G. Medioni},
  title = {{Exploring Local Context for Multi-target Tracking in Wide Area Aerial Surveillance}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2017},
  doi = {10.1109/wacv.2017.93},
}
Figure