Abstract
This paper presents our solution to the ACM MM challenge: Large-scale Human-centric Video Analysis in Complex Events; specifically, we focus on Track 3: Crowd Pose Tracking in Complex Events. While remarkable progress has been made in multi-pose training, tracking human poses in crowded and complex environments remains a significant challenge. We formulate the problem into several subproblems: first, a multi-object tracking method assigns human IDs to bounding boxes from a detection model; second, poses are generated for each ID-assigned box; finally, optical flow leverages temporal information to produce the final pose tracking result.
Introduction
Pose tracking—the task of identifying and following skeletal joints across video frames—is significantly harder in crowded environments where people frequently overlap, occlude one another, and move unpredictably. Standard methods often fail due to detection suppression (NMS) in dense crowds and temporal jitter caused by noise or partial occlusions. This work establishes a robust baseline for the HIE challenge by hardening the detection and tracking stages specifically for high-density scenes.
Methodology
The proposed pipeline follows a modular "top-down" approach consisting of four primary stages:
- Human Detection: Utilizing a Cascade R-CNN framework enhanced with CrowdDet. This incorporates Earth Mover’s Distance (EMD) loss to allow single proposals to predict multiple instances and "Set NMS" to prevent the suppression of valid overlapping detections.
- Human Tracking: An optimized DeepSORT implementation is used. A custom Re-Identification (ReID) model is trained to extract robust appearance features, maintaining identity consistency through occlusions.
- Pose Estimation: For each detection, heatmaps from two models—HRNet and SimpleNet (with FPN)—are fused to achieve precise joint localization.
- Temporal Smoothing: Optical Flow is employed to smooth bounding boxes and skeletal joints. By propagating joint positions from adjacent frames (bidirectional smoothing), the system eliminates jitter and fills gaps caused by brief occlusions.
Experimental Results
The system was evaluated on the Human in Events (HIE) dataset. A key finding was the critical importance of training data diversity.
Impact of Data and Modules
The inclusion of the CrowdHuman dataset provided a substantial performance boost, raising the Average Precision (AP) from ~61% to over 78%. This suggests that exposure to high-density overlapping instances is more impactful than architectural depth alone for this task.
Further refinements, including the use of "Set NMS" and EMD loss, allowed the detection model to reach 83.21% AP. The final integrated system achieved a Total_AP of 76.55 and a MOTA of 61.79 on the challenge test set.
Conclusion
This work demonstrates that a well-tuned combination of detection, tracking, and temporal smoothing components can effectively address the complexities of crowd pose tracking. While highly accurate, the multi-stage nature of the pipeline introduces computational overhead. Future research could explore end-to-end architectures to improve real-time performance and further refine handling of extreme occlusions.
Summary updated