Multi-task human analysis in still images: 2D/3D pose, depth map, and multi-part segmentation

Daniel Sánchez, Marc Oliu, Meysam Madadi, Xavier Baró, Sergio Escalera

Introduction

Analyzing human subjects in static images is a cornerstone of computer vision, encompassing tasks such as 2D and 3D joint localization, body part segmentation, and depth estimation. While these tasks are inherently related—for instance, knowing the boundaries of a limb should theoretically assist in locating its joints—multi-task learning (MTL) in this domain has historically been limited by a lack of datasets containing simultaneous, high-quality annotations for all these modalities.

This study systematically investigates the cross-task correlations between 2D pose, 3D pose, depth estimation, and body part segmentation. By leveraging the large-scale synthetic SURREAL dataset, the authors explore which task combinations yield synergistic benefits and which result in performance degradation, providing a roadmap for optimizing human pose and shape estimation models.

Methodology

To facilitate information sharing across different objectives, the researchers framed all four problems as pixel-level prediction tasks. 2D and 3D poses are represented via heatmaps (with 3D depth quantized into discrete bins), while depth estimation and body part segmentation are treated as pixel-wise classification problems.

The core architecture utilizes a Stacked Hourglass (SH) network. Unlike simple hard parameter sharing, the model assigns a dedicated SH stream to each task. These streams are interconnected via residual modules that fuse intermediate features, allowing the network to share semantic knowledge across tasks before producing final task-specific predictions.

Data: The SURREAL Dataset

The experiment relies on the SURREAL (Synthetic URban REAListic) dataset, which contains approximately 6.5 million frames. By using Motion Capture (MoCap) data to animate 3D human models against static backgrounds, SURREAL provides the dense, multi-modal ground truth necessary for training complex MTL architectures that real-world datasets currently lack.

Task Synergies and Conflicts

The study's primary contribution is the identification of specific task interactions that dictate performance gains:

  • The Role of Segmentation: Body part segmentation emerged as the most beneficial auxiliary task. By providing clear semantic outlines, it significantly enhances performance in depth estimation and 3D pose recovery.
3D body part segmentation provides semantic outlines that benefit auxiliary tasks.
  • The "Greedy" Nature of 3D Pose: Contrary to intuition, adding 3D pose estimation to a 2D pose or depth pipeline can actually hinder performance. Since 3D pose inherently contains 2D pose information, it tends to dominate the network's capacity without offering new regularizing features, sometimes leading to overfitting.
3D pose estimation, while comprehensive, can sometimes hinder the performance of simpler subtasks like 2D pose.
  • Depth and 2D Pose Synergy: These two tasks complement each other well; depth information helps resolve 2D ambiguities like self-occlusion, while 2D joint locations help the model better localize depth values.

Results and Performance

The multi-task approach demonstrated clear improvements over single-task baselines across several metrics:

  • Segmentation: Achieved a peak IoU of 68.16% when paired with 2D pose and depth.
  • 2D Pose: Reached a PCKh score of 97.01% in the four-task configuration, outperforming existing state-of-the-art models (90.8%).
  • Depth Estimation: Saw an 8% error reduction (RMSE of 4.04) when combined with segmentation and 3D pose.
  • 3D Pose: Achieved a Mean Joint Distance (MJD) of 56.99mm when paired with segmentation and depth, though this still trailed the current SOTA of 40.8mm.

Notably, the most significant gains were observed in body parts with high kinematic variability, such as wrists, ankles, and hands, where the shared features from multiple tasks helped resolve complex poses.

Conclusion

While multi-task learning offers a powerful path toward robust human analysis, this research highlights that not all tasks are mutually beneficial. The findings suggest that future architectures should carefully select auxiliary tasks—prioritizing segmentation and depth—and perhaps implement dynamic task weighting to prevent complex tasks from cannibalizing the shared representation space. Future work will likely involve extending these synthetic-trained models to real-world datasets through domain adaptation.

Summary updated

Abstract

While many individual tasks in the domain of human analysis have recently received an accuracy boost from deep learning approaches, multi-task learning has mostly been ignored due to a lack of data. New synthetic datasets are being released, filling this gap with synthetic generated data. In this work, we analyze four related human analysis tasks in still images in a multi-task scenario by leveraging such datasets. Specifically, we study the correlation of 2D/3D pose estimation, body part segmentation and full-body depth estimation. These tasks are learned via the well-known Stacked Hourglass module such that each of the task-specific streams shares information with the others. The main goal is to analyze how training together these four related tasks can benefit each individual task for a better generalization. Results on the newly released SURREAL dataset show that all four tasks benefit from the multi-task approach, but with different combinations of tasks: while combining all four tasks improves 2D pose estimation the most, 2D pose improves neither 3D pose nor full-body depth estimation. On the other hand 2D parts segmentation can benefit from 2D pose but not from 3D pose. In all cases, as expected, the maximum improvement is achieved on those human body parts that show more variability in terms of spatial distribution, appearance and shape, e.g. wrists and ankles.
BibTeX
@inproceedings{sanchez2019multi,
  author = {Daniel Sánchez and Marc Oliu and Meysam Madadi and Xavier Baró and Sergio Escalera},
  title = {{Multi-task human analysis in still images: 2D/3D pose, depth map, and multi-part segmentation}},
  booktitle = {FG},
  year = {2019},
  url = {https://openreview.net/forum?id=ijz8AjoRBiZu},
}
Figure