Abstract
PHALP (Predicting Human Appearance, Location, and Pose) is a robust tracking system that "lifts" 2D human detections into 3D space to maintain identity across occlusions and shot changes. By predicting a person's future 3D state—including their physical pose and appearance on a canonical 3D mesh—the system can track individuals even when they are temporarily invisible to the camera.
Introduction
Traditional multi-object tracking (MOT) usually happens in the 2D image plane. While this is computationally simpler, it suffers from a major conceptual flaw: in 2D, if someone walks behind a tree, they effectively cease to exist until they reappear. This "out of sight, out of mind" approach makes handling long-term occlusions and complex crowd interactions incredibly difficult.
Furthermore, 2D features like bounding box coordinates or pixel-based appearance are highly sensitive to camera movement. The authors argue that tracking should happen where the people actually are—in the 3D world. In 3D, an occluded person still has a location and a pose; they are just hidden from the current viewpoint.
Methods
The PHALP framework operates in three main stages: lifting to 3D, temporal prediction, and probabilistic association.
1. Lifting to 3D
For every 2D detection (using Mask R-CNN), the system uses a modified HMAR (Human Mesh Recovery with Appearance) model to estimate the person's 3D pose, their 3D location relative to the camera, and their 3D appearance mapped onto a canonical UV map of the human body.
2. Temporal Prediction
Instead of just looking at the current frame, PHALP maintains "tracklets" that aggregate data over time.
- Appearance: UV maps are averaged over time, allowing the model to "remember" what a person looks like from the front even if they are currently only showing their back.
- Pose: A Transformer-based model predicts future 3D poses (up to 12 frames ahead), helping the tracker anticipate movement.
- Location: The system uses linear regression to predict future 3D coordinates. Crucially, they use "nearness" (log inverse depth) to handle the non-linear way objects appear to move as they approach a lens.
3. Probabilistic Association
To match new detections to existing tracklets, the system calculates a similarity score based on the predicted 3D state. They model the noise in these attributes using specific distributions (Cauchy for pose/appearance and Exponential for location) and solve the final assignment using the Hungarian algorithm.
Key Insights and Ideas
3D Persistence and UV Mapping
By lifting to 3D, the model treats occlusion as a geometric property rather than a disappearance. The "state" of the person continues to evolve even when they aren't visible. Using a 3D mesh UV map is a game-changer for appearance. Unlike 2D crops, which change drastically as a person turns around, a UV map assigns pixels to specific body parts (e.g., "this pixel is the left elbow"), making the representation invariant to pose and viewpoint.
Handling Shot Changes
Because 3D pose and UV-mapped appearance are intrinsic to the person and independent of the camera, PHALP can maintain a track even after a movie cut. The system simply "drops" the location cue (which changes during a cut) and relies on pose and appearance to re-identify the person in the new shot.
Key Results
PHALP achieved state-of-the-art results across all tested benchmarks:
- PoseTrack: Reduced Identity Switches (IDs) to 541 (compared to 655 for the previous best) and achieved an IDF1 of 76.4.
- MuPoTS: Significantly outperformed competitors with a MOTA of 66.2.
- AVA: Demonstrated superior performance in tracking through camera cuts, achieving an IDF1 of 62.7.
Conclusion
PHALP demonstrates that lifting tracking to 3D space provides significant robustness against occlusions and camera changes. While dependent on the quality of initial detections and 3D lifting, the use of UV maps and temporal prediction of 3D states represents a major step forward in human tracking for complex video scenarios.
Summary updated