Track Initialization and Re-Identification for~3D Multi-View Multi-Object Tracking

Linh Van Ma, Tran Thien Dat Nguyen, Ba-Ngu Vo, Hyunsung Jang, Moongu JeonInformation Fusion 2024 · OpenReview

TL;DR

This paper introduces a novel 3D multi-object tracking (MOT) system that leverages 2D detections from multiple cameras to achieve robust tracking. It uniquely integrates track initialization, re-identification, and occlusion handling within a single Bayesian filtering framework, offering significant improvements in accuracy and adaptability, especially when camera configurations change dynamically.

What Problem Does This Paper Address?

Accurate 3D multi-object tracking (MOT) is crucial for understanding complex scenes, but it's a challenging task. Existing methods often rely on 3D object detectors, which are computationally expensive and require retraining when camera setups change. Furthermore, many 2D MOT approaches struggle with occlusions and re-identifying objects that reappear after being lost. This paper tackles these limitations by proposing a 3D MOT solution that uses only 2D detections from multiple cameras, eliminating the need for 3D detector retraining and seamlessly handling track initiation, termination, re-identification, and occlusions.

Methods

The core of this work is a Bayesian multi-object formulation that integrates various MOT functionalities into a single recursive filter. To overcome the computational intractability of exact Bayesian filtering, the authors propose an efficient approximation. Key components include:

  • Novel Dynamic and Measurement Models: These models jointly consider object kinematics, shape, visual features from multiple cameras, and occlusion.
  • Adaptive Birth Model: This model is crucial for automatically initiating new tracks and, importantly, for re-identifying tracks that have reappeared after being temporarily lost. It rectifies potential GLMB (Generalized Labeled Multi-Bernoulli) filter truncation issues that might otherwise discard these tracks.
  • Geometric Occlusion Model: This model uses the proportion of area overlap between 2D bounding boxes of objects in different camera views to estimate detection probabilities, effectively handling partial and full occlusions.
  • Multi-View GLMB Recursion: This forms the backbone of the tracking process, propagating the multi-object density over time. The paper details an approximation of this recursion that maintains linear complexity with respect to the number of detections.

Datasets

The proposed method was evaluated on two challenging multi-view MOT datasets:

  • WILDTRACK (WT) [13]: This dataset provides ground plane positions of objects.
  • Curtin Multi-Camera (CMC) [17]: This dataset offers more detailed information, including object height, width, and 3D positions, making it suitable for evaluating 3D trajectory tracking with extents. The CMC dataset includes sequences with varying object densities and scenarios with jumping and falling people.

Key Insights and Ideas

  • Re-identification without 3D Detectors: A significant breakthrough is the ability to re-identify lost tracks using only 2D detections and visual features, a capability not commonly found in existing 3D MOT systems that rely on 3D detectors.
  • Adaptive Birth for Robustness: The adaptive birth model is key to handling the dynamic nature of object appearances and disappearances, allowing the system to gracefully initiate new tracks and recover existing ones.
  • Geometric Occlusion for Accuracy: The novel occlusion model, based on 2D bounding box overlap, provides a more nuanced understanding of occlusions than previous methods, leading to more accurate detection probabilities.
  • Linear Complexity for Scalability: Achieving linear complexity in the number of detections is crucial for real-time applications and allows the system to scale to scenarios with many objects and cameras.
  • Camera Configuration Agnosticism: The method's ability to adapt to dynamic changes in camera configurations (adding, removing, or repositioning cameras) without retraining detectors is a major practical advantage.

Key Results

The proposed MV-GLMB-AB filter demonstrated superior performance across various metrics:

  • Accuracy: It achieved significantly higher MOTA (Multi-Object Tracking Accuracy) and IDF1 scores, and lower OSPA(2) errors compared to existing MV-GLMB and MS-GLMB filters on both WT and CMC datasets. For instance, on the WT dataset, the proposed filter achieved a MOTA of 36.1(2.60) compared to 16.2(1.79) for MV-GLMB.
  • Re-identification: Experiments simulating object disappearance and reappearance showed the filter's strong ability to re-identify lost tracks, with OSPA(2) errors remaining low even after cameras were turned off and back on.
  • Robustness to Camera Changes: The system showed comparable performance to baseline configurations even when cameras were added, removed, or repositioned on-the-fly, highlighting its adaptability.
  • Efficiency: The filter achieved real-time performance, with an average run-time of up to 112.34 FPS on a desktop CPU, making it suitable for practical applications.

Limitations and Future Work

  • Feature Stability: The paper notes that recall performance for re-identification is best when visual features are stable and unique. In scenarios with similar-looking objects or rapid appearance changes (e.g., pose changes), recall can be poor. This suggests that improving feature representation and robustness to appearance variations is an area for future work.
  • ID Switches: While the single-hypothesis approximation offers speed benefits, it can lead to an increase in ID switches, particularly in scenarios with high miss-detection rates. The authors suggest that an ad-hoc scheme to retain significant tracks from discarded hypotheses could mitigate this.
  • Ideal Detector Benchmarking: When benchmarked against ideal 3D detectors, the proposed filter's performance gaps were smaller than expected, suggesting that even with imperfect 2D detections, the tracking framework is highly effective. However, further investigation into improving the quality of 2D detections themselves could yield even better results.
  • Open Topic: The paper identifies improving features that are unique and time-invariant for re-identification as an ongoing challenge in computer vision.

Summary updated

Abstract

We propose a 3D multi-object tracking (MOT) solution using only 2D detections from monocular cameras, which automatically initiates/terminates tracks as well as resolves track appearance-reappearance and occlusions. Moreover, this approach does not require detector retraining when cameras are reconfigured but only the camera matrices of reconfigured cameras need to be updated. Our approach is based on a Bayesian multi-object formulation that integrates track initiation/termination, re-identification, occlusion handling, and data association into a single Bayes filtering recursion. However, the exact filter that utilizes all these functionalities is numerically intractable due to the exponentially growing number of terms in the (multi-object) filtering density, while existing approximations trade-off some of these functionalities for speed. To this end, we develop a more efficient approximation suitable for online MOT by incorporating object features and kinematics into the measurement model, which improves data association and subsequently reduces the number of terms. Specifically, we exploit the 2D detections and extracted features from multiple cameras to provide a better approximation of the multi-object filtering density to realize the track initiation/termination and re-identification functionalities. Further, incorporating a tractable geometric occlusion model based on 2D projections of 3D objects on the camera planes realizes the occlusion handling functionality of the filter. Evaluation of the proposed solution on challenging datasets demonstrates significant improvements and robustness when camera configurations change on-the-fly, compared to existing multi-view MOT solutions. The source code is publicly available at https://github.com/linh-gist/mv-glmb-ab.
BibTeX
@article{ma2024track,
  author = {Linh Van Ma and Tran Thien Dat Nguyen and Ba-Ngu Vo and Hyunsung Jang and Moongu Jeon},
  title = {{Track Initialization and Re-Identification for~3D Multi-View Multi-Object Tracking}},
  journal = {Information Fusion},
  year = {2024},
  url = {https://openreview.net/forum?id=orP1rUEZZb},
}
Figure