Tracking People by Predicting 3D Appearance, Location and Pose

Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, Jitendra Malik

Abstract

PHALP (Predicting Human Appearance, Location, and Pose) is a robust tracking system that "lifts" 2D human detections into 3D space to maintain identity across occlusions and shot changes. By predicting a person's future 3D state—including their physical pose and appearance on a canonical 3D mesh—the system can track individuals even when they are temporarily invisible to the camera.

Introduction

Traditional multi-object tracking (MOT) usually happens in the 2D image plane. While this is computationally simpler, it suffers from a major conceptual flaw: in 2D, if someone walks behind a tree, they effectively cease to exist until they reappear. This "out of sight, out of mind" approach makes handling long-term occlusions and complex crowd interactions incredibly difficult.

Furthermore, 2D features like bounding box coordinates or pixel-based appearance are highly sensitive to camera movement. The authors argue that tracking should happen where the people actually are—in the 3D world. In 3D, an occluded person still has a location and a pose; they are just hidden from the current viewpoint.

Methods

The PHALP framework operates in three main stages: lifting to 3D, temporal prediction, and probabilistic association.

Overview of the PHALP framework showing tracklet aggregation, prediction, and association with new detections.

1. Lifting to 3D

For every 2D detection (using Mask R-CNN), the system uses a modified HMAR (Human Mesh Recovery with Appearance) model to estimate the person's 3D pose, their 3D location relative to the camera, and their 3D appearance mapped onto a canonical UV map of the human body.

2. Temporal Prediction

Instead of just looking at the current frame, PHALP maintains "tracklets" that aggregate data over time.

  • Appearance: UV maps are averaged over time, allowing the model to "remember" what a person looks like from the front even if they are currently only showing their back.
  • Pose: A Transformer-based model predicts future 3D poses (up to 12 frames ahead), helping the tracker anticipate movement.
  • Location: The system uses linear regression to predict future 3D coordinates. Crucially, they use "nearness" (log inverse depth) to handle the non-linear way objects appear to move as they approach a lens.

3. Probabilistic Association

To match new detections to existing tracklets, the system calculates a similarity score based on the predicted 3D state. They model the noise in these attributes using specific distributions (Cauchy for pose/appearance and Exponential for location) and solve the final assignment using the Hungarian algorithm.

Key Insights and Ideas

3D Persistence and UV Mapping

By lifting to 3D, the model treats occlusion as a geometric property rather than a disappearance. The "state" of the person continues to evolve even when they aren't visible. Using a 3D mesh UV map is a game-changer for appearance. Unlike 2D crops, which change drastically as a person turns around, a UV map assigns pixels to specific body parts (e.g., "this pixel is the left elbow"), making the representation invariant to pose and viewpoint.

The HMAR model extracts appearance maps from single frames, which are then aggregated into a tracklet appearance map.

Handling Shot Changes

Because 3D pose and UV-mapped appearance are intrinsic to the person and independent of the camera, PHALP can maintain a track even after a movie cut. The system simply "drops" the location cue (which changes during a cut) and relies on pose and appearance to re-identify the person in the new shot.

Key Results

PHALP achieved state-of-the-art results across all tested benchmarks:

  • PoseTrack: Reduced Identity Switches (IDs) to 541 (compared to 655 for the previous best) and achieved an IDF1 of 76.4.
  • MuPoTS: Significantly outperformed competitors with a MOTA of 66.2.
  • AVA: Demonstrated superior performance in tracking through camera cuts, achieving an IDF1 of 62.7.

Conclusion

PHALP demonstrates that lifting tracking to 3D space provides significant robustness against occlusions and camera changes. While dependent on the quality of initial detections and 3D lifting, the use of UV maps and temporal prediction of 3D states represents a major step forward in human tracking for complex video scenarios.

Summary updated

Abstract

We present an approach for tracking people in monocular videos by predicting their future 3D representations. To achieve this, we first lift people to 3D from a single frame in a robust manner. This lifting includes information about the 3D pose of the person, their location in the 3D space, and the 3D appearance. As we track a person, we collect 3D observations over time in a tracklet representation. Given the 3D nature of our observations, we build temporal models for each one of the previous attributes. We use these models to predict the future state of the tracklet, including 3D appearance, 3D location, and 3D pose. For a future frame, we compute the similarity between the predicted state of a tracklet and the single frame observations in a probabilistic manner. Association is solved with simple Hungarian matching, and the matches are used to update the respective tracklets. We evaluate our approach on various benchmarks and report state-of-the-art results. Code and models are available at: https://brjathu.github.io/PHALP.
BibTeX
@inproceedings{rajasegaran2022tracking,
  author = {Jathushan Rajasegaran and Georgios Pavlakos and Angjoo Kanazawa and Jitendra Malik},
  title = {{Tracking People by Predicting 3D Appearance, Location and Pose}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00276},
}
Figure