Online Multiple Object Tracking Based on Open-Set Few-Shot Learning

Han-Ul Kim, Yeong Jun Koh, Chang-Su KimIEEE Access 2020 · DOI

This research presents a novel perspective on Online Multiple Object Tracking (MOT) by reframing it as an open-set few-shot classification problem. By developing a specialized embedding network called VOFNet, the authors achieve near real-time tracking speeds with a state-of-the-art balance between accuracy and computational efficiency.

The Tracking Challenge: Few-Shot and Open-Set

Multiple Object Tracking (MOT) involves identifying and following numerous targets—typically pedestrians—across a video sequence. "Online" trackers are particularly challenging because they must make decisions in real-time using only current and past frames, without the luxury of looking ahead.

The authors identify two primary hurdles in online MOT:

  1. The Few-Shot Problem: A tracker must learn what a specific person looks like based on only a few frames of data, even as their appearance changes due to lighting, pose, or perspective.
  2. The Open-Set Problem: The environment is dynamic. New objects enter the frame (unknown classes), and detectors often produce "ghost" detections or errors that should not be tracked.

Existing methods often rely on heavy motion models or complex data association schemes that slow down processing. This paper asks: Can we solve tracking by simply learning a better way to represent and compare object appearances?

Methodology: VOFNet

The core of the proposed system is VOFNet (Video Object embedding Network), which consists of two main components designed to handle spatial and temporal information.

Overview of the open-set few-shot classification framework for MOT.

VOF-CNN and VOF-RNN

  • VOF-CNN: This component uses an EfficientNet-B0 backbone to extract a 512-dimensional feature vector from image patches. It maps visual information into a "unit sphere" embedding space where similar-looking objects sit close together.
  • VOF-RNN: To handle temporal changes, a Recurrent Neural Network aggregates features from previous frames of a specific track. Crucially, it includes an attention mechanism that prevents the "representative feature" of a track from being corrupted by noisy data, such as when a person is momentarily occluded.

Tracking as Classification

The authors define the tracking task as a $(K + 2)$-way classification. For $K$ existing tracks, they add two "unknown" categories: $C_0$ (newly appearing objects) and $C_{-1}$ (detection errors). Data association is then handled via bipartite matching (the Hungarian algorithm), which links new detections to existing tracks based on their distance in the VOFNet embedding space.

Key Results and Performance

The model was evaluated on the MOT17 dataset, a standard benchmark for pedestrian tracking, using three different public detectors (DPM, FRCNN, and SDP) to ensure robustness.

The proposed tracker strikes an impressive balance between speed and precision:

  • Accuracy: Achieved 49.2 MOTA (Multiple Object Tracking Accuracy) on the MOT17 benchmark.
  • Speed: Processed video at 28.9 FPS, which is roughly 3.17 times faster than several recent high-performing competitors.
  • Ablation Success: The researchers proved that VOF-RNN outperformed simpler methods like "Moving Average," confirming that their temporal embedding strategy with attention is effective at maintaining target identity.

Limitations and Future Directions

A primary limitation is the reliance on the object detector. Because the tracker does not use a motion model (like a Kalman filter), it cannot "guess" where an object is if the detector fails to find it in a specific frame. If a person is hidden behind an obstacle for several frames and the detector misses them, the tracker may struggle to maintain the identity.

Future work aims to integrate motion modeling techniques to bridge the gap during detector failures, potentially combining robust appearance embeddings with predictive physics-based tracking.

Summary updated

BibTeX
@article{kim2020online,
  author = {Han-Ul Kim and Yeong Jun Koh and Chang-Su Kim},
  title = {{Online Multiple Object Tracking Based on Open-Set Few-Shot Learning}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3032252},
}
Figure