End-to-end Learning of Multi-sensor 3D Tracking by Detection

Davi Frossard, Raquel Urtasun

This paper from researchers at Uber ATG and the University of Toronto presents a unified framework for 3D multi-object tracking that bridges the gap between deep learning and classical optimization. By formulating tracking as a linear program and training the entire pipeline end-to-end, the authors demonstrate how to effectively fuse LIDAR and camera data for robust autonomous perception.

TL;DR

The authors propose an end-to-end trainable "tracking-by-detection" system that formulates multi-sensor data association as a linear program. By backpropagating through the optimization step using a structured hinge loss, the model learns to extract features that are specifically optimized for consistent 3D trajectory estimation.

What Problem Does This Paper Address?

Multi-target tracking (MTT) in autonomous driving is notoriously difficult due to object occlusions, sensor noise, and the complex motion of both the ego-vehicle and surrounding traffic. Traditionally, this is solved in two disjoint steps:

  1. Detection: Identifying objects in individual frames.
  2. Association: Linking detections across frames to form trajectories (often using hand-crafted distance metrics).

The "gap" here is that the detector doesn't know it's being used for tracking, and the tracker relies on fixed features that might not be optimal for matching. Furthermore, while LIDAR provides great spatial accuracy, it lacks the rich appearance cues of cameras. This paper seeks to solve both detection and association simultaneously in a single, differentiable 3D framework.

Methods

The core of the paper is a Deep Structured Model (DSM). Here is how it works:

  • The Formulation: Tracking is cast as an Integer Linear Program (ILP). The goal is to maximize a score based on four binary variables: whether a detection is a "true positive," whether two detections should be linked, and whether a trajectory starts or ends at a specific point.
  • The "Trick": While ILPs are generally NP-hard, the authors utilize the total unimodularity property of the flow constraints, allowing them to relax the problem into a standard Linear Program (LP) that can be solved exactly and efficiently.
  • Neural Scoring: Instead of hand-crafted costs, the costs for the LP are generated by neural networks:
  • Detection Network: A VGG16-based CNN that scores 3D proposals (from an MV3D backbone) by looking at camera patches.
  • Match Network: A Siamese network that compares appearance (camera) and spatial motion (LIDAR occupancy grids) between frames to decide if two detections belong to the same object.
  • End-to-End Learning: This is the most novel part. The authors use a structured hinge loss to train the system. This allows gradients to flow from the final tracking result all the way back to the initial feature extractors in the CNNs.

Datasets

The model was evaluated on the KITTI Tracking Benchmark, a standard for autonomous driving research.

  • It consists of 40 sequences (20 for training/validation, 20 for testing).
  • The dataset includes 3D annotations for vehicles, which the authors used to supervise their 3D tracking approach.

Key Insights and Ideas

  • End-to-End > Piecewise: The authors proved that training the detector and tracker together yields better results than training them separately. End-to-end training allows the model to find a better balance between precision and recall specifically for the task of maintaining long-term trajectories.
  • Differentiable Optimization: The paper demonstrates that you don't need to "approximate" tracking to make it deep-learning friendly. By using the structured hinge loss, you can treat a discrete optimization solver (the LP) as a layer in a neural network.
  • Multi-Sensor Synergy: The Siamese matching network uses a "skip-pooling" approach to combine low-level appearance (color) with high-level semantics. Meanwhile, the LIDAR branch uses occupancy grids in bird’s-eye view to handle ego-motion and spatial consistency, which is much more robust than 2D image-space motion.

Key Results

The model, referred to as DSM, achieved highly competitive results on the KITTI test set:

  • Precision and Consistency: It achieved the highest MOTP (Multi-Object Tracking Precision) at 83.42% and the highest percentage of Mostly Tracked (MT) objects at 60.00% among compared methods at the time.
  • Matching Accuracy: In a direct comparison of matching metrics, their "Deep Matching" approach had an error rate of only 3.27%, vastly outperforming traditional metrics like Cosine Similarity (29.66%) or Bounding Box Overlap (5.30%).
  • Trade-offs: End-to-end training significantly reduced ID-switches and fragmentations compared to piecewise training, though it tended to produce slightly shorter, more "certain" trajectories.

Limitations and Future Work

  • Long-term Occlusion: The current model focuses on matching detections in subsequent or near-subsequent frames. It still struggles with objects that disappear for long periods and then reappear.
  • Missing Detections: Because the tracker relies on a "tracking-by-detection" paradigm, if the detector fails to produce a proposal for a frame, the trajectory is likely to break.
  • Complexity: While the LP relaxation is efficient, solving a large-scale global optimization for very long sequences in real-time remains a computational challenge.

The authors suggest that future work will focus on extending the temporal window to handle more complex occlusion scenarios and improving the robustness to missing detections.

Summary updated

Abstract

In this paper we propose a novel approach to tracking by detection that can exploit both cameras as well as LIDAR data to produce very accurate 3D trajectories. Towards this goal, we formulate the problem as a linear program that can be solved exactly, and learn convolutional networks for detection as well as matching in an end-to-end manner. We evaluate our model in the challenging KITTI dataset and show very competitive results.
BibTeX
@inproceedings{frossard2018end,
  author = {Davi Frossard and Raquel Urtasun},
  title = {{End-to-end Learning of Multi-sensor 3D Tracking by Detection}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=hrrELgQrxL},
}

Loading related content...

Figure