Are Pose Estimators Ready for the Open World? STAGE: A GenAI Toolkit for Auditing 3D Human Pose Estimators

Nikita Kister, István Sárándi, Jiayi Wang, Anna Khoreva, Gerard Pons-Moll

TL;DR

This paper introduces STAGE, a novel toolkit that uses generative AI to create synthetic image pairs differing by a single attribute (like clothing or location). This allows for the first time a controlled audit of 3D human pose estimators, revealing significant performance drops due to common real-world variations, questioning their readiness for deployment.

What Problem Does This Paper Address?

Deploying 3D human pose estimators in safety-critical applications (like autonomous vehicles or robotics) requires them to be robust to a wide range of real-world conditions. Current benchmarks, often based on real images, struggle to provide controlled comparisons where only a single attribute (e.g., weather, clothing, gender) varies. This makes it impossible to isolate and quantify how specific factors impact pose estimation accuracy. The authors aim to bridge this gap by creating a tool that can generate precisely controlled synthetic data for auditing these estimators.

Methods

The core of STAGE is a novel generative AI model, CN-3DPose, which creates realistic human images with precise 3D pose control. Unlike previous methods, CN-3DPose can condition on 3D pose information (represented by depth maps, semantic encodings, and 2D skeletons) and text prompts to generate diverse and coherent images. The authors also propose a new evaluation strategy: they generate pairs of images that are identical except for one target attribute (e.g., "man wearing a t-shirt" vs. "man wearing a parka"). By comparing the pose estimation error on these pairs, they can quantify the sensitivity of an estimator to that specific attribute. A key metric introduced is the Percentage of Degraded Poses (PDP), which measures how often an estimator's error increases significantly when an attribute changes.

Datasets

  • Training Data: The CN-3DPose model was trained on a combination of existing datasets: AGORA, HUMBI, SHHQ, COCO, and human scan renderings.
  • Benchmark Generation: STAGE generates custom benchmarks for auditing. The authors created benchmarks for various attributes including clothing, texture, location (indoor/outdoor), and protected attributes (gender, age, ethnicity). They also created a synthetic replica of the 3DPW benchmark for evaluation.

Key Insights and Ideas

  • Real-world variations are a major challenge: Even state-of-the-art pose estimators, which perform well on standard benchmarks, show significant performance degradation when faced with common real-world attribute changes.
  • Clothing and texture are highly impactful: Attributes related to body appearance, particularly clothing items (like coats) and textures, have the most significant negative impact on pose estimation accuracy.
  • Indoor locations can be trickier than expected: Contrary to intuition, some indoor environments with occlusions (e.g., "bar," "restaurant") pose greater challenges than many outdoor scenes.
  • Protected attributes matter, but less than appearance: While estimators are less sensitive to protected attributes like ethnicity compared to clothing, gender and age do show a noticeable impact, highlighting potential fairness issues.
  • Large-scale, diverse training is crucial for robustness: Models trained on larger, more diverse datasets (especially those with varied appearance) tend to be more robust.
  • Architectural choices can improve robustness: Beyond data, smart architectural designs (like PyMAF-X's iterative feedback loop) can also contribute to more robust pose estimation.

Key Results

  • STAGE successfully generates diverse, pose-aligned images that differ by single attributes, enabling controlled experiments.
  • The authors audited several popular pose estimators (SPIN, PARE, PyMAF-X, BEDLAM-CLIFF, etc.) and found that all of them exhibit significant sensitivity to attribute changes. For instance, clothing changes could lead to up to a 30% increase in degraded predictions for some estimators.
  • Models trained on STAGE data achieved performance comparable to models trained on the highly curated, but labor-intensive, CG dataset BEDLAM, demonstrating STAGE's effectiveness and efficiency.
  • Qualitative results (Figure 4) visually demonstrate how simple attribute changes (e.g., "shirt" to "parka") can cause noticeable errors in pose predictions.

Limitations and Future Work

  • Single-person, single-image focus: STAGE currently focuses on single-person, single-image pose estimation and does not address interpersonal interactions or video sequences.
  • Generative model imperfections: While CN-3DPose is advanced, current generative models are not perfect and can still have limitations.
  • Future work: The authors suggest extending STAGE to handle more complex scenarios like human-object interactions and video. They also envision integrating more advanced generative models as they become available. The code and data will be released to facilitate further research.

Summary updated

Abstract

For safety-critical applications, it is crucial to audit 3D human pose estimators before deployment. Will the system break down if the weather or the clothing changes? Is it robust regarding gender and age? To answer these questions and more, we need controlled studies with images that differ in a single attribute, but real benchmarks cannot provide such pairs. We thus present STAGE, a GenAI data toolkit for auditing 3D human pose estimators. For STAGE, we develop the first GenAI image creator with accurate 3D pose control and propose a novel evaluation strategy to isolate and quantify the effects of single factors such as gender, ethnicity, age, clothing, location, and weather. Enabled by STAGE, we generate a series of benchmarks to audit, for the first time, the sensitivity of popular pose estimators towards such factors. Our results show that natural variations can severely degrade pose estimator performance, raising doubts about their readiness for open-world deployment. We aim to highlight these robustness issues and establish STAGE as a benchmark to quantify them.
BibTeX
@inproceedings{kister2026are,
  author = {Nikita Kister and István Sárándi and Jiayi Wang and Anna Khoreva and Gerard Pons-Moll},
  title = {{Are Pose Estimators Ready for the Open World? STAGE: A GenAI Toolkit for Auditing 3D Human Pose Estimators}},
  booktitle = {3DV},
  year = {2026},
  doi = {10.1109/3dv69130.2026.00089},
}
Figure