A Simple Baseline for Pose Tracking in Videos of Crowded Scenes

Li Yuan, Shuning Chang, Ziyuan Huang, Yichen Zhou, Yunpeng Chen, Xuecheng Nie, Francis E. H. Tay, Jiashi Feng, Shuicheng Yan

Abstract

This paper presents our solution to the ACM MM challenge: Large-scale Human-centric Video Analysis in Complex Events; specifically, we focus on Track 3: Crowd Pose Tracking in Complex Events. While remarkable progress has been made in multi-pose training, tracking human poses in crowded and complex environments remains a significant challenge. We formulate the problem into several subproblems: first, a multi-object tracking method assigns human IDs to bounding boxes from a detection model; second, poses are generated for each ID-assigned box; finally, optical flow leverages temporal information to produce the final pose tracking result.

Introduction

Pose tracking—the task of identifying and following skeletal joints across video frames—is significantly harder in crowded environments where people frequently overlap, occlude one another, and move unpredictably. Standard methods often fail due to detection suppression (NMS) in dense crowds and temporal jitter caused by noise or partial occlusions. This work establishes a robust baseline for the HIE challenge by hardening the detection and tracking stages specifically for high-density scenes.

Methodology

The proposed pipeline follows a modular "top-down" approach consisting of four primary stages:

  1. Human Detection: Utilizing a Cascade R-CNN framework enhanced with CrowdDet. This incorporates Earth Mover’s Distance (EMD) loss to allow single proposals to predict multiple instances and "Set NMS" to prevent the suppression of valid overlapping detections.
  2. Human Tracking: An optimized DeepSORT implementation is used. A custom Re-Identification (ReID) model is trained to extract robust appearance features, maintaining identity consistency through occlusions.
  3. Pose Estimation: For each detection, heatmaps from two models—HRNet and SimpleNet (with FPN)—are fused to achieve precise joint localization.
  4. Temporal Smoothing: Optical Flow is employed to smooth bounding boxes and skeletal joints. By propagating joint positions from adjacent frames (bidirectional smoothing), the system eliminates jitter and fills gaps caused by brief occlusions.
The proposed multi-stage pipeline for crowd pose tracking.

Experimental Results

The system was evaluated on the Human in Events (HIE) dataset. A key finding was the critical importance of training data diversity.

Impact of Data and Modules

The inclusion of the CrowdHuman dataset provided a substantial performance boost, raising the Average Precision (AP) from ~61% to over 78%. This suggests that exposure to high-density overlapping instances is more impactful than architectural depth alone for this task.

Ablation study showing the impact of different training datasets on validation performance.

Further refinements, including the use of "Set NMS" and EMD loss, allowed the detection model to reach 83.21% AP. The final integrated system achieved a Total_AP of 76.55 and a MOTA of 61.79 on the challenge test set.

Conclusion

This work demonstrates that a well-tuned combination of detection, tracking, and temporal smoothing components can effectively address the complexities of crowd pose tracking. While highly accurate, the multi-stage nature of the pipeline introduces computational overhead. Future research could explore end-to-end architectures to improve real-time performance and further refine handling of extreme occlusions.

Summary updated

Abstract

This paper presents our solution to ACM MM challenge: Large-scale Human-centric Video Analysis in Complex Events\cite{lin2020human}; specifically, here we focus on Track3: Crowd Pose Tracking in Complex Events. Remarkable progress has been made in multi-pose training in recent years. However, how to track the human pose in crowded and complex environments has not been well addressed. We formulate the problem as several subproblems to be solved. First, we use a multi-object tracking method to assign human ID to each bounding box generated by the detection model. After that, a pose is generated to each bounding box with ID. At last, optical flow is used to take advantage of the temporal information in the videos and generate the final pose tracking result.
BibTeX
@inproceedings{yuan2020simple,
  author = {Li Yuan and Shuning Chang and Ziyuan Huang and Yichen Zhou and Yunpeng Chen and Xuecheng Nie and Francis E. H. Tay and Jiashi Feng and Shuicheng Yan},
  title = {{A Simple Baseline for Pose Tracking in Videos of Crowded Scenes}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=M1GKmhpQpOJ},
}
Figure