Abstract
This paper addresses the problem of online tracking and classification of multiple objects in an image sequence. Our proposed solution is to first track all objects in the scene without relying on object-specific prior knowledge, which in other systems can take the form of hand-crafted features or user-based track initialization. We then classify the tracked objects with a fast-learning image classifier based on a shallow convolutional neural network architecture and demonstrate that object recognition improves when combined with object state information from the tracking algorithm. We argue that by transferring the use of prior knowledge from the detection and tracking stages to the classification stage, we can design a robust, general-purpose object recognition system. Our biologically inspired implementation adaptively learns the shape and motion of tracked objects. Evaluation on the Neovision2 Tower benchmark demonstrates that our approach is competitive with state-of-the-art systems while providing additional practical advantages through its generality.
Introduction: Moving Beyond Tracking-by-Detection
Most modern multi-object tracking (MOT) systems rely on a "tracking-by-detection" framework, requiring specialized detectors (e.g., for persons or cars) to initialize tracks. This creates a "chicken and egg" problem: a system can only track what it already knows how to detect. If it encounters a new object type or lacks a massive labeled dataset, the system fails. Furthermore, highly tuned detectors are often brittle when faced with domain shifts or environmental clutter.
This research proposes a "track-everything-then-classify" paradigm, flipping the traditional approach. By emulating the separate "what" and "where" processing streams found in biological vision, the system autonomously tracks all salient objects in a scene—including previously unseen types—without requiring object-specific prior knowledge during the tracking phase.
A Biologically Inspired Dual-Stream Architecture
The architecture splits the workload into two distinct streams that reunite to perform final classification.
The "Where" Stream: CACTuS-FL
The tracking engine, CACTuS-FL, uses a Shape Estimating Filter (SEF). This is a hierarchical Bayesian model that recursively learns an object’s position, velocity, and probabilistic shape. Instead of searching for specific classes, it identifies "salient blobs" using a bank of unsupervised convolutional filters. It employs a "winner-take-more" competitive mechanism where multiple SEFs compete to "explain" different parts of the image, effectively tracking everything in the scene, including background clutter.
The "What" Stream: S-CNN and Integration
The "What" stream utilizes a Shallow Convolutional Neural Network (S-CNN) for fast, frame-by-frame classification of image patches identified by the trackers. The core innovation lies in the integration stage: an ensemble of Single Hidden Layer Feedforward Networks (SLFNs) combines the S-CNN's class predictions with the "state information" from the tracker (such as bounding box dimensions and movement angle). This metadata helps the system distinguish between actual targets and tracked clutter.
Experimental Evaluation
The system was evaluated using the Neovision2 Tower benchmark, featuring top-down aerial footage of a campus. The dataset includes Car, Person, and Cyclist classes, along with a "Clutter" class for background objects. Unlike many datasets focusing only on motion, Neovision2 requires tracking both moving and stationary targets.
Key Results
- Competitive Performance: The system achieved results competitive with state-of-the-art neuromorphic and computer vision algorithms.
- Integration Boost: Combining the "what" and "where" streams led to a massive performance gain, providing an improvement of 0.4 in Average WNMOTDA (Weighted Normalized Multiple Object Thresholded Detection Accuracy) compared to using the classifier alone.
- Class Leaders: The approach achieved top scores for the "Car" and "Cyclist" classes within the comparison group.
Discussion and Future Work
A primary insight of this work is that tracking "everything" allows the system to treat background clutter as persistent objects rather than new detections in every frame, significantly reducing false positives. Furthermore, physical proportions and movement patterns (the "where" data) are powerful discriminators; for instance, distinct movement patterns help distinguish a blurry person from a car.
A current limitation involves object extent. Because the system tracks salient blobs without initial class knowledge, it can struggle with groups (e.g., a crowd tracked as one large "Person" blob). Future work will focus on refining object splitting and merging and optimizing the shallow architectures for lower-latency performance on edge hardware.
Summary updated