Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking

Sarthak Sharma, Junaid Ahmed Ansari, J. Krishna Murthy, K. Madhava Krishna

TL;DR

This paper introduces novel, geometry- and shape-based cost functions for online multi-object tracking using only monocular camera data. These intuitive and computationally efficient costs significantly improve tracking accuracy, even outperforming state-of-the-art methods when combined with a simple Hungarian assignment algorithm.

What Problem Does This Paper Address?

Accurate multi-object tracking (MOT) is crucial for autonomous driving and urban scene understanding. The dominant "tracking-by-detection" paradigm relies heavily on the data association phase, which is notoriously challenging due to factors like missed detections, occlusions, and complex object interactions. Existing methods often use handcrafted features (like bounding box positions or color histograms) or complex, non-intuitive cost functions. This paper aims to bridge this gap by developing simple, yet effective, pairwise costs that leverage readily available 3D geometric and shape information from monocular images, making MOT more robust and accurate without requiring complex machinery.

Methods

The core contribution is the introduction of new pairwise cost functions for multi-object tracking. These costs are designed to be integrated into any existing tracking-by-detection framework.

  • 3D-2D Cost: This cost leverages the inherent geometry of road scenes. By backprojecting detected bounding boxes from frame t to frame t+1 using estimated 3D position and uncertainty (derived from monocular depth estimation), it defines a search area in the next frame. The cost is then based on the overlap between this predicted area and actual detections in frame t+1. This significantly reduces the search space for potential matches.
  • 3D-3D Cost: To mitigate the limitations of 2D overlap measurements, this cost directly compares the 3D volumes of object hypotheses in 3D space. It backprojects detections to 3D and measures their overlap, providing a more robust cue.
  • Appearance Cost: This cost utilizes features extracted from a stacked-hourglass Convolutional Neural Network (CNN) architecture. This CNN is trained to localize discriminative keypoints, and its activation maps serve as a feature descriptor for each detection, capturing appearance similarity.
  • Shape and Pose Cost: This novel cost leverages recent work on estimating 3D object shape (represented by deformation coefficients) and pose (axis-angle vector) from monocular images. It directly compares the shape and pose parameters of object hypotheses, providing a strong cue for distinguishing between similar-looking objects or tracking objects with varying orientations.

These costs are combined linearly, with weights determined through cross-validation. The overall framework uses a standard tracking-by-detection pipeline, feeding these costs into a bipartite matching algorithm (specifically, the Hungarian algorithm for a simple two-frame association).

Datasets

The primary dataset used for evaluation is the KITTI Tracking benchmark. The authors utilize both the training and testing splits. They also mention dividing the training set into four splits for cross-validation to tune the weights of their cost functions. No new benchmarks were introduced.

Key Insights and Ideas

  • Leveraging Monocular 3D Geometry is Powerful: The paper convincingly demonstrates that even without stereo vision or LiDAR, monocular 3D cues (shape, pose, and geometric projection) are highly effective for improving multi-object tracking accuracy.
  • Simplicity Can Trump Complexity: By combining intuitive, geometry-based costs with a simple two-frame Hungarian assignment, the authors achieve state-of-the-art results, suggesting that the quality of the cost functions is more critical than the complexity of the association algorithm itself.
  • Complementary Cues are Key: The ablation studies show that each proposed cost (3D-2D, 3D-3D, appearance, shape/pose) contributes to performance improvement, and their combination yields the best results. This highlights the benefit of diverse cues.
  • Shape and Pose Matter in Challenging Scenarios: The shape and pose costs become particularly relevant in situations like intersections or roundabouts where object appearance might be similar, but their 3D orientation and form differ significantly.

Key Results

  • State-of-the-Art Performance: The proposed method achieves state-of-the-art results on the KITTI Tracking benchmark.
  • On the training set, they report MOTA of 91.4% and MOTP of 89.83% using a simple two-frame Hungarian assignment.
  • On the online evaluation server (test set), they outperform the next best competitor by a margin of 6%, achieving MOTA of 84.24%, MOTP of 85.73%, MT of 73.23%, and ML of 2.77%.
  • Detector Agnosticism: The approach shows consistent improvement across different object detectors (RRC and SubCNN), demonstrating its generality.
  • Efficiency: The costs are described as easy to implement and computable in real-time, crucial for online tracking.

Limitations and Future Work

  • ID Switches and Fragmentations: As with most online trackers, the method can suffer from ID switches and track fragmentations, which are typical of greedy, two-frame association approaches.
  • Sophisticated Trackers: The authors suggest that incorporating their proposed costs into more sophisticated tracking frameworks (e.g., network flow-based methods) would likely further reduce ID switches and fragmentations and lead to even higher performance.
  • Future Work: The paper implies that exploring more complex data association schemes with their costs is a promising direction.

Summary updated

Abstract

This paper introduces geometry and object shape and pose costs for multi-object tracking in urban driving scenarios. Using images from a monocular camera alone, we devise pairwise costs for object tracks, based on several 3D cues such as object pose, shape, and motion. The proposed costs are agnostic to the data association method and can be incorporated into any optimization framework to output the pairwise data associations. These costs are easy to implement, can be computed in real-time, and complement each other to account for possible errors in a tracking-by-detection framework. We perform an extensive analysis of the designed costs and empirically demonstrate consistent improvement over the state-of-the-art under varying conditions that employ a range of object detectors, exhibit a variety in camera and object motions, and, more importantly, are not reliant on the choice of the association framework. We also show that, by using the simplest of associations frameworks (two-frame Hungarian assignment), we surpass the state-of-the-art in multi-object-tracking on road scenes. More qualitative and quantitative results can be found at https://junaidcs032.github.io/Geometry_ObjectShape_MOT/.
BibTeX
@inproceedings{sharma2018beyond,
  author = {Sarthak Sharma and Junaid Ahmed Ansari and J. Krishna Murthy and K. Madhava Krishna},
  title = {{Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=uIK2MN5dkE},
}
Figure