Track Everything: Limiting Prior Knowledge in Online Multi-Object Recognition

Sebastien C. Wong, Victor Stamatescu, Adam Gatt, David A. Kearney, Ivan Lee, Mark D. McDonnell

TL;DR

This paper proposes a "track-everything-then-classify" paradigm for online multi-object recognition, flipping the traditional "detect-then-track" approach on its head. By emulating the separate "what" and "where" processing streams found in biological vision, the system can autonomously track all salient objects in a scene—including previously unseen types—without requiring object-specific prior knowledge during the tracking phase.

What Problem Does This Paper Address?

Most modern multi-object tracking (MOT) systems rely on a "tracking-by-detection" framework. This means you first run a specialized detector (e.g., a "person detector" or "car detector") and then link those detections over time.

This creates a "chicken and egg" problem: you can only track what you already know how to detect. If you encounter a new object type or lack a massive labeled dataset for training, the system fails. Furthermore, highly tuned detectors are often brittle when faced with domain shifts or environmental clutter. The authors address this lack of generality by moving object-specific knowledge out of the tracking stage and into a later classification stage.

Methods

The authors introduce a biologically inspired architecture that splits the workload into two streams:

  1. The "Where" Stream (CACTuS-FL): This is the tracking engine. It uses a Shape Estimating Filter (SEF), a hierarchical Bayesian model that recursively learns an object’s position, velocity, and—crucially—its probabilistic shape. Instead of looking for "cars," it looks for "salient blobs" using a bank of unsupervised convolutional filters. It employs a "winner-take-more" competitive mechanism where multiple SEFs compete to "explain" different parts of the image, effectively tracking everything in the scene, including background clutter.
  2. The "What" Stream (S-CNN): A Shallow Convolutional Neural Network provides fast, frame-by-frame classification of the image patches identified by the trackers.
  3. The Integration (SLFN Ensemble): The core innovation is how these streams reunite. An ensemble of Single Hidden Layer Feedforward Networks (SLFNs) combines the S-CNN's class predictions with the "state information" from the tracker (like the object's bounding box dimensions and movement angle). This helps the system distinguish between actual targets (cars, people) and tracked clutter.

Datasets

The system was evaluated using the Neovision2 Tower benchmark, a challenging dataset featuring top-down aerial-like footage of a campus.

  • Classes: Car, Person, Cyclist, and a "Clutter" class for background objects.
  • Training: 15 image sequences (010–024) were used for offline training of the S-CNN and SLFNs.
  • Testing: 12 sequences (001–023) were used for the final evaluation.
  • Unique Aspect: Unlike many datasets that focus only on moving objects, Neovision2 requires detecting and tracking both moving and stationary targets.

Key Insights and Ideas

  • Explaining Away the Background: By tracking "everything," the system treats background clutter as just another set of objects to be tracked. This prevents clutter from being "rediscovered" as a new detection in every frame, which significantly reduces false positives.
  • The Power of Metadata: The researchers found that knowing how an object moves and its physical proportions (the "where" data) significantly boosts classification accuracy. For example, a "person" patch might look like a "car" patch in a single blurry frame, but their distinct movement patterns and aspect ratios help the SLFN ensemble make the right call.
  • Limiting Prior Knowledge: By using unsupervised feature learning (via Convolutional Restricted Boltzmann Machines) for the tracking filters, the system avoids the bias inherent in hand-crafted features.

Key Results

  • Competitive Performance: The system achieved results competitive with state-of-the-art neuromorphic and computer vision algorithms on the Neovision2 benchmark.
  • Integration Boost: Combining the "what" and "where" streams led to a massive improvement in performance. Specifically, the integration provided a gain of 0.4 in Average WNMOTDA (Weighted Normalized Multiple Object Thresholded Detection Accuracy) compared to using the classifier alone.
  • Class Leaders: The approach achieved the top scores for the "Car" and "Cyclist" classes within the comparison group.

Limitations and Future Work

The authors identify a primary limitation regarding object extent. Because the system tracks salient blobs without knowing what they are initially, it can struggle with groups. For instance, a crowd of people walking together might be tracked by a single SEF as one large "object." While the classifier correctly identifies the blob as "Person," the resulting bounding box is much larger than the ground-truth boxes for the individuals, leading to a lower accuracy score in standard evaluations.

Future work could involve refining how the system handles object splitting and merging, or further optimizing the "shallow" network architectures for even lower-latency performance on edge hardware.

Summary updated

Abstract

This paper addresses the problem of online tracking and classification of multiple objects in an image sequence. Our proposed solution is to first track all objects in the scene without relying on object-specific prior knowledge, which in other systems can take the form of hand-crafted features or user-based track initialization. We then classify the tracked objects with a fast-learning image classifier, that is based on a shallow convolutional neural network architecture and demonstrate that object recognition improves when this is combined with object state information from the tracking algorithm. We argue that by transferring the use of prior knowledge from the detection and tracking stages to the classification stage, we can design a robust, general purpose object recognition system with the ability to detect and track a variety of object types. We describe our biologically inspired implementation, which adaptively learns the shape and motion of tracked objects, and apply it to the Neovision2 Tower benchmark data set, which contains multiple object types. An experimental evaluation demonstrates that our approach is competitive with the state-of-the-art video object recognition systems that do make use of object-specific prior knowledge in detection and tracking, while providing additional practical advantages by virtue of its generality.
BibTeX
@article{wong2017track,
  author = {Sebastien C. Wong and Victor Stamatescu and Adam Gatt and David A. Kearney and Ivan Lee and Mark D. McDonnell},
  title = {{Track Everything: Limiting Prior Knowledge in Online Multi-Object Recognition}},
  journal = {IEEE Trans. Image Process.},
  year = {2017},
  url = {https://openreview.net/forum?id=SotKgCIGDG},
}

Loading related content...

Figure