Human Pose Estimation in Videos

Dong Zhang, Mubarak ShahICCV 2015 · OpenReview

Abstract

In this paper, we present a method to estimate a sequence of human poses in unconstrained videos. In contrast to the commonly employed graph optimization framework, which is NP-hard and needs approximate solutions, we formulate this problem into a unified two stage tree-based optimization problem for which an efficient and exact solution exists. Although the proposed method finds an exact solution, it does not sacrifice the ability to model the spatial and temporal constraints between body parts in the video frames, indeed it even models the symmetric parts better than the existing methods. The proposed method is based on two main ideas: 'Abstraction' and 'Association' to enforce the intra-and inter-frame body part constraints respectively without inducing extra computational complexity to the polynomial time solution. Using the idea of 'Abstraction', a new concept of 'abstract body part' is introduced to model not only the tree based body part structure similar to existing methods, but also extra constraints between symmetric parts. Using the idea of 'Association', the optimal tracklets are generated for each abstract body part, in order to enforce the spatiotemporal constraints between body parts in adjacent frames. Finally, a sequence of the best poses is inferred from the abstract body part tracklets through the tree-based optimization. We evaluated the proposed method on three publicly available video based human pose estimation datasets, and obtained dramatically improved performance compared to the state-of-the-art methods.

Introduction

Estimating human poses in video is a complex balancing act between achieving spatial accuracy within individual frames and maintaining temporal consistency across the sequence. Most current approaches model this as a graph optimization problem. However, when researchers attempt to model all relevant constraints—such as the relationship between symmetric limbs (e.g., left vs. right ankles) and the movement of parts over time—the resulting graph becomes "loopy." In computational terms, these cycles make the problem NP-hard, forcing a reliance on approximate solutions that are often slow or sub-optimal.

Furthermore, standard tree-based models, while computationally efficient, frequently suffer from "double counting." This occurs when the algorithm confuses symmetric limbs due to their visual similarity, incorrectly placing both estimated labels on the same physical limb.

Methodology

The proposed framework addresses these challenges by transforming the complex optimization problem into a unified two-stage tree-based process using two core concepts: Abstraction and Association.

The proposed pipeline: (a) Input video, (b) N-Best pose hypotheses, (c) Real body part hypotheses, (d) Abstract body part hypotheses, (e) Tracklet generation, and (f) Final tree-based pose estimation.

Abstraction (Intra-frame)

To eliminate cycles within a single frame's relational graph, the authors introduce the concept of an "abstract body part." Instead of treating symmetric limbs as separate nodes, they are combined into a single unit. This coupling forces the model to select a pair of locations that represent both limbs simultaneously, effectively resolving the double-counting problem.

Association (Inter-frame)

To handle the temporal dimension, the method generates "tracklets"—short, consistent paths of these abstract parts across video frames. Because each abstract part is tracked independently at this stage, no temporal cycles are introduced into the final graph.

Tree-based Optimization

The final sequence of poses is inferred by selecting the best tracklets through a tree-based optimization. Because the resulting graph is a structure without loops, it can be solved exactly and in polynomial time using dynamic programming. This ensures that the method finds the mathematically optimal solution efficiently.

Experimental Results

The method was evaluated on three established benchmarks: the Outdoor Pose Dataset, Human Eva-I, and the N-Best Dataset. These datasets present significant challenges, including self-occlusion, cluttered backgrounds, and diverse pose configurations.

Qualitative results comparing the proposed method ("Ours") against N-Best and Symmetric Tracklet ("Sym. Trk.") baselines across various video sequences.

The proposed method significantly outperformed existing state-of-the-art techniques:

  • Outdoor Dataset: Achieved a 0.92 Probability of a Correct Pose (PCP), compared to 0.86 for the previous best method. It also showed a much lower Keypoint Localization Error (KLE) of 0.36 vs. 0.58.
  • Human Eva-I: Maintained high accuracy (0.92 PCP) while significantly reducing localization errors.
  • Efficiency: Despite finding an exact solution, the Matlab implementation is fast, processing approximately 1.1 seconds per frame on a standard desktop.

Conclusion

By strategically "collapsing" nodes through Abstraction and linking them via Association, this framework turns a difficult optimization problem into an efficient, solvable one. While the method provides an exact solution, it remains dependent on the initial N-Best pose generator; if the correct limb candidates are missing from the initial pool, the optimization cannot recover them. Future research may apply this Abstraction and Association formulation to other complex computer vision problems.

Summary updated

Abstract

In this paper, we present a method to estimate a sequence of human poses in unconstrained videos. In contrast to the commonly employed graph optimization framework, which is NP-hard and needs approximate solutions, we formulate this problem into a unified two stage tree-based optimization problem for which an efficient and exact solution exists. Although the proposed method finds an exact solution, it does not sacrifice the ability to model the spatial and temporal constraints between body parts in the video frames, indeed it even models the symmetric parts better than the existing methods. The proposed method is based on two main ideas: 'Abstraction' and 'Association' to enforce the intra-and inter-frame body part constraints respectively without inducing extra computational complexity to the polynomial time solution. Using the idea of 'Abstraction', a new concept of 'abstract body part' is introduced to model not only the tree based body part structure similar to existing methods, but also extra constraints between symmetric parts. Using the idea of 'Association', the optimal tracklets are generated for each abstract body part, in order to enforce the spatiotemporal constraints between body parts in adjacent frames. Finally, a sequence of the best poses is inferred from the abstract body part tracklets through the tree-based optimization. We evaluated the proposed method on three publicly available video based human pose estimation datasets, and obtained dramatically improved performance compared to the state-of-the-art methods.
BibTeX
@inproceedings{zhang2015human,
  author = {Dong Zhang and Mubarak Shah},
  title = {{Human Pose Estimation in Videos}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=rkbXKgzd-B},
}
Figure