Deep Fully-Connected Part-Based Models for Human Pose Estimation.

Rodrigo de Bem, Anurag Arnab, Stuart Golodetz, Michael Sapienza, Philip H. S. TorrACML 2018 · OpenReview

Abstract

We propose a 2D multi-level appearance representation of the human body in RGB images, spatially modelled using a fully-connected graphical model. The appearance model is based on a CNN body part detector that learns to jointly predict joints, limbs, and the whole body. This hierarchy provides an inductive bias that improves localization, especially for ambiguous joints. To enforce global spatial consistency, we employ a fully-connected Conditional Random Field (CRF) where inference is implemented as a Recurrent Neural Network (RNN). This implementation allows for end-to-end training of the entire system, enabling the model to learn optimal spatial relationships directly from data.

Introduction

Estimating human pose from a single RGB image remains a significant challenge in computer vision due to the body's high degree of articulation, frequent occlusions, and the inherent ambiguity of projecting 3D poses onto a 2D plane. While modern "bottom-up" methods have made strides in detecting individual joints, they often lack the global context necessary to resolve complex scenarios, such as unusual body orientations or cluttered backgrounds where objects may mimic human parts. This paper, presented at ACML 2018, addresses these issues by combining a multi-level CNN detector with a fully-connected spatial model.

Methodology

The proposed framework integrates local appearance detection with global spatial reasoning through a two-stage pipeline.

The overall framework: A multi-level part detector generates initial heatmaps, which are then refined by a spatial model to ensure global consistency.

Multi-Level Appearance Representation

Instead of focusing solely on joint coordinates, the model represents the human body at three distinct levels of granularity: joints (e.g., elbow, knee), rigid parts (e.g., forearm, thigh), and the whole body. This hierarchical approach is based on the insight that larger structures like limbs are often easier to identify than specific joints. By detecting the forearm, the network gains a strong inductive bias for locating the associated wrist and elbow.

The Cascade CNN architecture for multi-level detection, showing shared features and progressive estimation of joints (J), rigid parts (R), and the whole body (B).

The architecture uses a Cascade CNN where features are shared across modules. To train this multi-level system without requiring additional manual labeling, the authors developed a weak-supervision method that automatically generates ground-truth heatmaps for limbs and bodies based on existing joint annotations.

Fully-Connected CRF as an RNN

To ensure that the detected parts form a valid human pose, the system employs a Conditional Random Field (CRF). Unlike traditional "tree-structured" models that only connect adjacent joints in the skeleton, this model is fully connected—every part is connected to every other part. This allows the model to learn complex dependencies, such as the spatial relationship between the left and right hips, which are not directly connected in a skeletal tree.

The key innovation is implementing the Mean-Field inference algorithm for the CRF as a Recurrent Neural Network (RNN). This makes the spatial model fully differentiable, allowing the CNN detector and the CRF to be trained jointly. During training, the model learns the mean displacement and covariance between all parts, effectively discovering the "rules" of human anatomy from the data.

Experiments and Results

The model was evaluated on two major benchmarks: the MPII Human Pose Dataset and the Leeds Sports Pose (LSP) Dataset.

  • Performance: The system achieved competitive results, with 90.2% Mean [email protected] on LSP and 88.1% Mean [email protected] on MPII.
  • Robustness: Qualitative analysis showed that the model is significantly more robust than standard bottom-up detectors when dealing with unusual poses (e.g., upside-down figures) and cluttered environments.
  • Ablation Study: The inclusion of the spatial model (SM) consistently improved accuracy, particularly in resolving "double-counting" errors where a detector might otherwise identify two left feet or two right hands.

Limitations and Future Work

While effective, the model's primary drawback is computational complexity. The addition of the CRF-based spatial model increases inference time from approximately 1.5 seconds to 7.5 seconds per image in a multi-scale testing environment. Furthermore, the current use of Gaussian kernels for pairwise relations may limit the model's ability to handle multi-modal distributions where a limb could plausibly be in two very different positions.

Future research directions include extending the framework to multi-person pose estimation and incorporating additional data modalities, such as color and depth information, into the CRF kernels to further improve spatial reasoning.

Summary updated

Abstract

We propose a 2D multi-level appearance representation of the human body in RGB images, spatially modelled using a fully-connected graphical model. The appearance model is based on a CNN body part d...
BibTeX
@inproceedings{bem2018deep,
  author = {Rodrigo de Bem and Anurag Arnab and Stuart Golodetz and Michael Sapienza and Philip H. S. Torr},
  title = {{Deep Fully-Connected Part-Based Models for Human Pose Estimation.}},
  booktitle = {ACML},
  year = {2018},
  url = {https://openreview.net/forum?id=ZZxU_cpeJr},
}
Figure