Abstract
In this paper we propose a novel approach to tracking by detection that can exploit both cameras as well as LIDAR data to produce very accurate 3D trajectories. Towards this goal, we formulate the problem as a linear program that can be solved exactly, and learn convolutional networks for detection as well as matching in an end-to-end manner. We evaluate our model in the challenging KITTI dataset and show very competitive results.
Introduction
Researchers at Uber ATG and the University of Toronto have developed a unified framework for 3D multi-object tracking (MOT) that bridges the gap between deep learning and classical optimization. By formulating tracking as a linear program and training the entire pipeline end-to-end, the authors demonstrate how to effectively fuse LIDAR and camera data for robust autonomous perception.
The Challenge of Multi-Target Tracking
Multi-target tracking in autonomous driving is notoriously difficult due to object occlusions, sensor noise, and the complex motion of both the ego-vehicle and surrounding traffic. Traditionally, this problem is solved in two disjoint steps:
- Detection: Identifying objects in individual frames.
- Association: Linking detections across frames to form trajectories, often using hand-crafted distance metrics.
The primary issue with this decoupled approach is that the detector is unaware of its role in tracking, and the tracker relies on fixed features that may not be optimal for matching. Furthermore, while LIDAR provides high spatial accuracy, it lacks the rich appearance cues available from cameras. This research seeks to solve both detection and association simultaneously within a single, differentiable 3D framework.
Methods: The Deep Structured Model
The core of this approach is the Deep Structured Model (DSM), which casts tracking as an Integer Linear Program (ILP). The objective is to maximize a score based on four binary variables: whether a detection is a "true positive," whether two detections should be linked, and whether a trajectory starts or ends at a specific point.
While ILPs are generally NP-hard, the authors exploit the total unimodularity property of the flow constraints. This allows the problem to be relaxed into a standard Linear Program (LP) that can be solved exactly and efficiently. The most significant innovation is the use of a structured hinge loss, which enables gradients to flow from the final tracking result back to the initial feature extractors, allowing for true end-to-end learning.
Neural Scoring and Matching
Instead of relying on hand-crafted costs, the costs for the LP are generated by specialized neural networks:
- Detection Network: A VGG16-based CNN that scores 3D proposals (derived from an MV3D backbone) by analyzing camera patches.
- Match Network: A Siamese network that compares appearance (camera) and spatial motion (LIDAR occupancy grids) between frames to determine if two detections belong to the same object.
The Siamese matching network utilizes a "skip-pooling" approach to combine low-level appearance features with high-level semantics. Simultaneously, the LIDAR branch uses occupancy grids in a bird’s-eye view to handle ego-motion and spatial consistency, providing a more robust solution than 2D image-space motion alone.
Datasets and Evaluation
The model was evaluated on the KITTI Tracking Benchmark, a standard for autonomous driving research. The dataset consists of 40 sequences (20 for training/validation and 20 for testing) and includes 3D annotations for vehicles, which were used to supervise the 3D tracking approach.
Key Insights and Results
The authors demonstrated that training the detector and tracker together yields superior results compared to piecewise training. End-to-end training allows the model to find a better balance between precision and recall specifically for maintaining long-term trajectories.
The model, referred to as DSM, achieved highly competitive results on the KITTI test set:
- Precision and Consistency: It achieved the highest MOTP (Multi-Object Tracking Precision) at 83.42% and the highest percentage of Mostly Tracked (MT) objects at 60.00% among compared methods at the time.
- Matching Accuracy: In a direct comparison, the "Deep Matching" approach had an error rate of only 3.27%, vastly outperforming traditional metrics like Cosine Similarity (29.66%) or Bounding Box Overlap (5.30%).
- Trade-offs: End-to-end training significantly reduced ID-switches and fragmentations, though it tended to produce slightly shorter, more "certain" trajectories.
Limitations and Future Work
Despite its strengths, the model has limitations. It currently focuses on matching detections in near-subsequent frames, struggling with objects that disappear for long periods. Additionally, as a "tracking-by-detection" system, it is sensitive to detector failures; if a proposal is missed, the trajectory may break. Finally, while the LP relaxation is efficient, scaling global optimization for very long sequences remains a computational challenge. Future work will focus on extending the temporal window and improving robustness to missing detections.
Summary updated