Integrating Pose and Mask Predictions for Multi-person in Videos

Miran Heo, Sukjun Hwang, Seoung Wug Oh, Joon-Young Lee, Seon Joo KimWorkshops @ CVPR 2022 · OpenReview

Abstract

In real-world applications for video editing, humans are arguably the most important objects. When editing videos of humans, the efficient tracking of fine-grained masks and body joints is the fundamental requirement. In this paper, we propose a simple and efficient system for jointly tracking pose and segmenting high-quality masks for all humans in the video. We design a pipeline that globally tracks pose and locally segments fine-grained masks. Specifically, CenterTrack is first employed to track human poses by viewing the whole scene, and then the proposed local segmentation network leverages the pose information as a powerful query to carry out high-quality segmentation. Furthermore, we adopt a highly light-weight MLP-Mixer layer within the segmentation network that can efficiently propagate the query pose throughout the region of interest with minimal overhead. For the evaluation, we collect a new benchmark called KineMask which includes various appearances and actions. The experimental results demonstrate that our method has superior fine-grained segmentation performance. Moreover, it runs at 33 fps, achieving a great balance of speed and accuracy compared to the prevailing online Video Instance Segmentation methods.
BibTeX
@inproceedings{heo2022integrating,
  author = {Miran Heo and Sukjun Hwang and Seoung Wug Oh and Joon-Young Lee and Seon Joo Kim},
  title = {{Integrating Pose and Mask Predictions for Multi-person in Videos}},
  booktitle = {Workshops @ CVPR},
  year = {2022},
  url = {https://openreview.net/forum?id=L1NMS6tMP6Y},
}

Loading related content...

Figure