Towards a Principled Integration of Multi-Camera Re-Identification and Tracking through Optimal Bayes Filters

Lucas Beyer, Stefan Breuers, Vitaly Kurin, Bastian Leibe

Abstract

With the rise of end-to-end learning through deep learning, person detectors and re-identification (ReID) models have recently become very strong. Multi-camera multi-target (MCMT) tracking has not fully gone through this transformation yet. We intend to take another step in this direction by presenting a theoretically principled way of integrating ReID with tracking formulated as an optimal Bayes filter. This conveniently side-steps the need for data-association and opens up a direct path from full images to the core of the tracker. While the results are still sub-par, we believe that this new, tight integration opens many interesting research opportunities and leads the way towards full end-to-end tracking from raw pixels.

Introduction

Most multi-target multi-camera (MTMC) tracking systems follow a traditional "tracking-by-detection" paradigm: 1) detect people in frames, 2) extract features for re-identification, and 3) solve a complex "data association" problem to link detections into tracks.

This "detection boundary" acts as a bottleneck. Detections are discrete and binary, throwing away rich, probabilistic information present in raw pixels. Furthermore, data association—deciding which detection belongs to which person—is computationally expensive and prone to error. This research explores moving toward a "pixels-to-tracks" approach where the tracker maintains a continuous belief about a person's location across the entire image.

Methodology: Tracking as a Histogram Filter

The core of this work is the formulation of tracking as a Histogram Filter, a specific implementation of an optimal Bayes filter.

The ReID Measurement Model

Instead of using a discrete detector, the authors utilize a ReID CNN (a modified "LuNet"). By converting the network's fully-connected layers into $1 \times 1$ convolutions, the model can be run over an entire image to produce a low-resolution "embedding map."

The ReID measurement model generates probability maps for specific identities.

For a given track, the person's known embedding is compared to the embedding map of the current frame, resulting in a distance map. A softmin operation then transforms this into a probability map—essentially a "heat map" of where that specific person is likely to be.

Predict and Update Steps

To account for motion, the tracker takes the probability map from the previous frame and convolves it with a Gaussian kernel (the Predict Step). This spreads the probability mass, representing movement and increasing uncertainty. In the Update Step, the predicted map (prior) and the new ReID measurement map are multiplied element-wise and re-normalized to yield the "posterior" probability of the person's location. This principled approach follows Bayesian inference to propagate uncertainty rather than relying on heuristic bounding box matching.

Experimental Setup

The method was evaluated on the DukeMTMC dataset, which features 8 static cameras and nearly 3,000 person identities. It is a high-resolution, high-frame-rate (60 FPS) dataset, posing a significant challenge compared to older benchmarks. Evaluation was performed on a subset (trainval-mini), ensuring the ReID model was trained on identities not present in the validation set to test generalization.

Key Insights and Ideas

By treating a deep ReID model as a noisy sensor in a classic filtering framework, several advantages emerge:

  • Sidestepping Data Association: Since the measurement model is person-specific (asking "where is Person A?"), the need for global data association algorithms like the Hungarian algorithm disappears.
  • Detection-less Tracking: The tracker maintains a full probability distribution over the image, allowing it to represent multi-modal beliefs, such as a person being behind one of two possible pillars.
The posterior belief and resulting tracks generated by the Bayes filter.

However, a major challenge is background noise. ReID models, typically trained on cropped images, often "fire" on background textures that resemble clothing, leading to high-entropy noise when applied to full images.

Results

While the theoretical framework is elegant, quantitative results on standard metrics were lower than traditional baselines:

  • Baselines: A standard batch-tracking method (BIPCC) achieved a MOTA of 76.7, while a simple Nearest-Neighbor Kalman Filter (NN-KF) reached 36.6.
  • Integrated Model: The initial integrated model performed poorly (MOTA -13.0) due to high False Positives.
  • Entropy Gating: Implementing a mechanism to discard high-entropy measurements (where the model is "confused" by the background) improved the MOTA to 14.2.

Limitations and Future Work

The current system relies on ground truth initialization and cannot yet "discover" new people entering a scene. Furthermore, the ReID model lacks a sense of "personness," sometimes matching identities to background elements like grass. Future work involves integrating a continuous "detector" to distinguish people from background and addressing drift caused by significant appearance changes (e.g., lighting). Finally, the authors suggest that standard metrics like MOTA may unfairly penalize systems providing rich, continuous probability maps, indicating a need for new probabilistic tracking evaluation measures.

Summary updated

Abstract

With the rise of end-to-end learning through deep learning, person detectors and re-identification (ReID) models have recently become very strong. Multi-camera multi-target (MCMT) tracking has not fully gone through this transformation yet. We intend to take another step in this direction by presenting a theoretically principled way of integrating ReID with tracking formulated as an optimal Bayes filter. This conveniently side-steps the need for data-association and opens up a direct path from full images to the core of the tracker. While the results are still sub-par, we believe that this new, tight integration opens many interesting research opportunities and leads the way towards full end-to-end tracking from raw pixels.
BibTeX
@inproceedings{beyer2017towards,
  author = {Lucas Beyer and Stefan Breuers and Vitaly Kurin and Bastian Leibe},
  title = {{Towards a Principled Integration of Multi-Camera Re-Identification and Tracking through Optimal Bayes Filters}},
  booktitle = {Workshops @ CVPR},
  year = {2017},
  url = {https://openreview.net/forum?id=Hk-DkRZubS},
}
Figure