Track Everything: Limiting Prior Knowledge in Online Multi-Object Recognition

Sebastien C. Wong, Victor Stamatescu, Adam Gatt, David A. Kearney, Ivan Lee, Mark D. McDonnell

Abstract

This paper addresses the problem of online tracking and classification of multiple objects in an image sequence. Our proposed solution is to first track all objects in the scene without relying on object-specific prior knowledge, which in other systems can take the form of hand-crafted features or user-based track initialization. We then classify the tracked objects with a fast-learning image classifier based on a shallow convolutional neural network architecture and demonstrate that object recognition improves when combined with object state information from the tracking algorithm. We argue that by transferring the use of prior knowledge from the detection and tracking stages to the classification stage, we can design a robust, general-purpose object recognition system. Our biologically inspired implementation adaptively learns the shape and motion of tracked objects. Evaluation on the Neovision2 Tower benchmark demonstrates that our approach is competitive with state-of-the-art systems while providing additional practical advantages through its generality.

Introduction: Moving Beyond Tracking-by-Detection

Most modern multi-object tracking (MOT) systems rely on a "tracking-by-detection" framework, requiring specialized detectors (e.g., for persons or cars) to initialize tracks. This creates a "chicken and egg" problem: a system can only track what it already knows how to detect. If it encounters a new object type or lacks a massive labeled dataset, the system fails. Furthermore, highly tuned detectors are often brittle when faced with domain shifts or environmental clutter.

This research proposes a "track-everything-then-classify" paradigm, flipping the traditional approach. By emulating the separate "what" and "where" processing streams found in biological vision, the system autonomously tracks all salient objects in a scene—including previously unseen types—without requiring object-specific prior knowledge during the tracking phase.

A Biologically Inspired Dual-Stream Architecture

The architecture splits the workload into two distinct streams that reunite to perform final classification.

The proposed dual-stream architecture separating 'Where' (tracking) and 'What' (recognition) processing.

The "Where" Stream: CACTuS-FL

The tracking engine, CACTuS-FL, uses a Shape Estimating Filter (SEF). This is a hierarchical Bayesian model that recursively learns an object’s position, velocity, and probabilistic shape. Instead of searching for specific classes, it identifies "salient blobs" using a bank of unsupervised convolutional filters. It employs a "winner-take-more" competitive mechanism where multiple SEFs compete to "explain" different parts of the image, effectively tracking everything in the scene, including background clutter.

The Shape Estimating Filter (SEF) hierarchical Bayesian model for tracking object state and shape.

The "What" Stream: S-CNN and Integration

The "What" stream utilizes a Shallow Convolutional Neural Network (S-CNN) for fast, frame-by-frame classification of image patches identified by the trackers. The core innovation lies in the integration stage: an ensemble of Single Hidden Layer Feedforward Networks (SLFNs) combines the S-CNN's class predictions with the "state information" from the tracker (such as bounding box dimensions and movement angle). This metadata helps the system distinguish between actual targets and tracked clutter.

Experimental Evaluation

The system was evaluated using the Neovision2 Tower benchmark, featuring top-down aerial footage of a campus. The dataset includes Car, Person, and Cyclist classes, along with a "Clutter" class for background objects. Unlike many datasets focusing only on motion, Neovision2 requires tracking both moving and stationary targets.

Key Results

  • Competitive Performance: The system achieved results competitive with state-of-the-art neuromorphic and computer vision algorithms.
  • Integration Boost: Combining the "what" and "where" streams led to a massive performance gain, providing an improvement of 0.4 in Average WNMOTDA (Weighted Normalized Multiple Object Thresholded Detection Accuracy) compared to using the classifier alone.
  • Class Leaders: The approach achieved top scores for the "Car" and "Cyclist" classes within the comparison group.

Discussion and Future Work

A primary insight of this work is that tracking "everything" allows the system to treat background clutter as persistent objects rather than new detections in every frame, significantly reducing false positives. Furthermore, physical proportions and movement patterns (the "where" data) are powerful discriminators; for instance, distinct movement patterns help distinguish a blurry person from a car.

A current limitation involves object extent. Because the system tracks salient blobs without initial class knowledge, it can struggle with groups (e.g., a crowd tracked as one large "Person" blob). Future work will focus on refining object splitting and merging and optimizing the shallow architectures for lower-latency performance on edge hardware.

Summary updated

Abstract

This paper addresses the problem of online tracking and classification of multiple objects in an image sequence. Our proposed solution is to first track all objects in the scene without relying on object-specific prior knowledge, which in other systems can take the form of hand-crafted features or user-based track initialization. We then classify the tracked objects with a fast-learning image classifier that is based on a shallow convolutional neural network architecture and demonstrate that object recognition improves when this is combined with object state information from the tracking algorithm. We argue that by transferring the use of prior knowledge from the detection and tracking stages to the classification stage we can design a robust, general purpose object recognition system with the ability to detect and track a variety of object types. We describe our biologically inspired implementation, which adaptively learns the shape and motion of tracked objects, and apply it to the Neovision2 Tower benchmark data set, which contains multiple object types. An experimental evaluation demonstrates that our approach is competitive with state-of-the-art video object recognition systems that do make use of object-specific prior knowledge in detection and tracking, while providing additional practical advantages by virtue of its generality.
BibTeX
@article{wong2017track,
  author = {Sebastien C. Wong and Victor Stamatescu and Adam Gatt and David A. Kearney and Ivan Lee and Mark D. McDonnell},
  title = {{Track Everything: Limiting Prior Knowledge in Online Multi-Object Recognition}},
  journal = {IEEE Trans. Image Process.},
  year = {2017},
  url = {https://openreview.net/forum?id=SotKgCIGDG},
}
Figure