Multi-scale Aggregation R-CNN for 2D Multi-person Pose Estimation

Gyeongsik Moon, Ju Yong Chang, Kyoung Mu Lee

TL;DR

This paper introduces the Multi-scale Aggregation R-CNN (MSA R-CNN), a novel approach for 2D multi-person pose estimation that unifies human detection and keypoint localization into a single model. By effectively leveraging multi-scale feature information through new MS-RoIAlign and MS-KpsNet modules, MSA R-CNN achieves state-of-the-art performance among single-model methods while significantly reducing computational cost compared to existing top-down approaches.

What Problem Does This Paper Address?

Accurately identifying the keypoints (like joints) of multiple people in a 2D image is a complex task. Existing "top-down" methods, which first detect humans and then estimate their poses, have achieved impressive results by utilizing multi-scale information from images. This multi-scale information is vital because it captures both fine-grained details around individual joints and broader contextual information about the entire person. However, these methods suffer from a major drawback: they require a separate, computationally expensive human detection model to pre-process the image before feeding it to the pose estimation network. This leads to a significant overall computational burden. The paper aims to bridge this gap by developing a method that effectively utilizes multi-scale information for accurate pose estimation while simultaneously reducing computational complexity by integrating detection and pose estimation into a single, unified model.

Methods

The core innovation of MSA R-CNN lies in its unified architecture and its specialized modules for handling multi-scale features:

  • Unified Model: Unlike previous top-down methods that use two distinct networks (one for detection, one for pose estimation), MSA R-CNN combines both tasks within a single model. This allows for shared feature extraction, drastically reducing computation.
  • Multi-scale RoIAlign (MS-RoIAlign): This module is designed to extract features for human proposals from multiple levels of the feature pyramid network (FPN) backbone, rather than a single scale. It then aggregates these multi-scale features. This contrasts with the original RoIAlign, which selects features from a single scale based on proposal size, potentially missing crucial information.
  • Multi-scale Keypoint Head Network (MS-KpsNet): This network takes the aggregated multi-scale features from MS-RoIAlign and uses a U-Net-like architecture with downsampling and upsampling layers, incorporating residual skip connections. This design allows it to effectively capture and combine both local (fine-grained) and global (contextual) information for accurate heatmap prediction of each keypoint.

The backbone network used is a Feature Pyramid Network (FPN) based on ResNet [12] or ResNeXt [34], which is standard practice for extracting multi-scale features.

Datasets

  • MS COCO keypoint detection dataset [18]: This is the primary dataset used for both training and evaluation. The paper specifies using the training set for training and the validation and test-dev sets for evaluation.

Key Insights and Ideas

  • Unification is Key to Efficiency: Combining human detection and pose estimation into a single model is a highly effective strategy for reducing computational cost without sacrificing accuracy. This challenges the necessity of separate, resource-intensive detection pipelines.
  • Multi-scale Feature Aggregation is Crucial: Simply using features from a single scale (even if it's the "best" scale for a given proposal) is suboptimal. Aggregating features from multiple scales (both fine-grained local details and coarse global context) significantly improves pose estimation accuracy.
  • Dedicated Modules for Multi-scale Handling: The proposed MS-RoIAlign and MS-KpsNet are not just minor tweaks; they are fundamental architectural changes that directly address the limitations of previous methods in leveraging multi-scale information.

Key Results

  • State-of-the-Art for Single Models: MSA R-CNN achieved the best performance among single model-based methods on the MS COCO test-dev set.
  • Comparable to Separated Models: The proposed method achieved results comparable to those of more complex, separated model-based methods, but with significantly less computation.
  • Performance Gains: The paper reports an improvement of 6.2 AP over the baseline Mask R-CNN model through the combined effect of its proposed modules. Specifically, the MS-KpsNet contributed a 1.5 AP increase, and the MS-RoIAlign contributed a 0.9 AP increase.
  • Computational Efficiency: Table 4 shows that MSA R-CNN requires less GPU memory and has a faster testing time compared to other advanced methods, even those with simpler architectures. For instance, it requires 48% longer running time and 70% more GPU memory for CPN-50 to achieve similar accuracy.

Limitations and Future Work

  • Computational Resource Constraints: The authors acknowledge that their unified model's computational resource limitations (e.g., GPU memory) prevent them from easily adopting even deeper backbone networks, which are a common strategy for performance boosting in other methods.
  • Future Work: While not explicitly detailed, the success of their multi-scale aggregation approach suggests that further exploration of more sophisticated feature aggregation techniques and potentially even deeper backbone networks within their unified framework could lead to further performance gains. The release of their code aims to facilitate reproduction and further research.

Summary updated

Abstract

Multi-person pose estimation from a 2D image is challenging because it requires not only keypoint localization but also human detection. In state-of-the-art top-down methods, multi-scale information is a crucial factor for the accurate pose estimation because it contains both of local information around the keypoints and global information of the entire person. Although multi-scale information allows these methods to achieve the state-of-the-art performance, the top-down methods still require a huge amount of computation because they need to use an additional human detector to feed the cropped human image to their pose estimation model. To effectively utilize multi-scale information with the smaller computation, we propose a multi-scale aggregation R-CNN (MSA R-CNN). It consists of multi- scale RoIAlign block (MS-RoIAlign) and multi-scale keypoint head network (MS-KpsNet) which are designed to effectively utilize multi-scale information. Also, in contrast to previous top-down methods, the MSA R-CNN performs human detection and keypoint localization in a single model, which results in reduced computation. The proposed model achieved the best performance among single model-based methods and its results are comparable to those of separated model-based methods with a smaller amount of computation on the publicly available 2D multi-person keypoint localization dataset.
BibTeX
@inproceedings{moon2019multi,
  author = {Gyeongsik Moon and Ju Yong Chang and Kyoung Mu Lee},
  title = {{Multi-scale Aggregation R-CNN for 2D Multi-person Pose Estimation}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=ltxxeuhNCv33},
}
Figure