A Survey on Model Based Approaches for 2D and 3D Visual Human Pose Recovery

Xavier Perez-Sala, Sergio Escalera, Cecilio Angulo, Jordi Gonzàlez

TL;DR

This paper provides a comprehensive survey and taxonomy of model-based approaches for 2D and 3D human pose recovery. It categorizes existing methods into five key modules – appearance, viewpoint, spatial relations, temporal consistency, and behavior – and offers a methodological comparison to highlight current trends, advantages, and drawbacks in this challenging computer vision problem.

What Problem Does This Paper Address?

Estimating the precise 3D pose of a human from images or video is a fundamental yet incredibly difficult problem in computer vision. This is due to numerous factors: the vast number of degrees of freedom in the human body, variations in clothing and self-occlusion, changes in viewpoint, and dynamic, often cluttered, real-world environments. While significant progress has been made over the last 40 years, achieving robust human pose recovery in uncontrolled settings remains an open challenge. This paper aims to bring order to the field by providing a structured overview and comparative analysis of the state-of-the-art model-based approaches.

Methods

The core contribution of this paper is its proposed taxonomy, which organizes model-based human pose recovery methods into five distinct modules:

  • Appearance: How visual information (e.g., intensity, color, texture, depth) is used to describe and detect human bodies or their parts. This includes descriptors like HOG, SIFT, and depth-based features, as well as classification techniques.
  • Viewpoint: How the camera's perspective and the human's orientation are handled, crucial for reducing ambiguities in 3D pose estimation. This is categorized into discrete (classification) and continuous (estimation) approaches.
  • Spatial Models: How the relationships between different body parts are represented. This includes "ensembles of parts" (like pictorial structures and poselets) and "structure models" (like kinematic chains and 3D skeletons).
  • Temporal Models: How information from video sequences is leveraged to improve pose estimation. This is primarily divided into tracking (maintaining pose consistency over time) and motion models (capturing the dynamics of human movement).
  • Behavior: How activity or context information is incorporated to refine pose estimation or analyze actions. This is a less explored area but is seen as a promising avenue for improving generalization.

The paper then systematically reviews existing literature, categorizing numerous works according to this taxonomy and performing a methodological comparison.

Datasets

The paper does not introduce new datasets. Instead, it references various datasets used in the cited works, often mentioning them in the context of specific actions or scenarios (e.g., dancing, walking, golf swings, skiing, skating). The table in Figure 7 indicates when a dataset was used for training or as a specific context.

Key Insights and Ideas

  • The "Appearance" module is the most stable: Edge-based descriptors like HOG are widely used and have proven effective for body part detection, forming a strong foundation for many approaches.
  • "Spatial Models" are diverse and evolving: While ensembles of parts (e.g., pictorial structures) have been dominant, newer approaches using multi-view trees and grammars are showing promise in handling occlusions and variability. Structure models are better suited for 3D accuracy due to physical constraints.
  • "Temporal Models" are heavily skewed towards tracking: When video is available, tracking is the primary focus for leveraging temporal information, often improving 2D methods by reducing viewpoint-induced nonlinearities. Motion models, while potentially powerful for reducing search space, are less explored.
  • "Viewpoint" and "Behavior" are less explored but critical: Joint viewpoint and 3D pose estimation is challenging but offers a way to reduce estimation ambiguities. Similarly, incorporating "behavior" or context is emerging as a trend to enhance generalization and scene understanding.
  • The future lies in integration: The paper strongly suggests that future trends will involve combining knowledge of the global scene, object interactions, and human activity with pose and motion analysis for more robust and generalized human pose recovery.

Key Results

The paper is a survey and taxonomy, not an experimental paper reporting new quantitative results. Its "results" are the insights derived from comparing existing methods based on the proposed taxonomy. The table in Figure 7 provides a comparative overview of various state-of-the-art approaches across the five modules, highlighting which modules each work addresses and their general approach (e.g., discrete vs. continuous viewpoint, ensembles of parts vs. structure models). The discussion section summarizes that while some methods achieve excellent results (e.g., [10]), there's no single benchmark for direct performance comparison across all approaches.

Limitations and Future Work

  • Lack of a Common Benchmark: The authors explicitly state that a major limitation is the absence of a standardized benchmark for comparing 2D and 3D pose estimation approaches, as well as joint pose and behavior estimation. This makes direct performance comparisons difficult.
  • Motion Models Under-Explored: While temporal models are dominated by tracking, motion models have not been deeply explored, despite their potential to reduce the search space for 3D human pose estimation.
  • Behavior and Context Integration: While some works use datasets organized by actions, simultaneous estimation of behavior and pose, or using activity estimation to refine pose, is not yet common.
  • Future Trends: The paper points towards combining global scene knowledge, object context, and activity information with pose and motion analysis. This suggests a move towards more holistic "ambient intelligence" systems. Further research is also suggested for general motion models that can handle noisy input data and integrate with body part detectors.

Summary updated

Abstract

Human Pose Recovery has been studied in the field of Computer Vision for the last 40 years. Several approaches have been reported, and significant improvements have been obtained in both data representation and model design. However, the problem of Human Pose Recovery in uncontrolled environments is far from being solved. In this paper, we define a general taxonomy to group model based approaches for Human Pose Recovery, which is composed of five main modules: appearance, viewpoint, spatial relations, temporal consistence, and behavior. Subsequently, a methodological comparison is performed following the proposed taxonomy, evaluating current SoA approaches in the aforementioned five group categories. As a result of this comparison, we discuss the main advantages and drawbacks of the reviewed literature.
BibTeX
@inproceedings{perezsala2014survey,
  author = {Xavier Perez-Sala and Sergio Escalera and Cecilio Angulo and Jordi Gonzàlez},
  title = {{A Survey on Model Based Approaches for 2D and 3D Visual Human Pose Recovery}},
  booktitle = {Sensors},
  year = {2014},
  url = {https://openreview.net/forum?id=gQn51ol5Ibi},
}
Figure