Revisiting Reliability in the Reasoning-based Pose Estimation Benchmark

Junsu Kim, Naeun Kim, Jaeho Lee, Incheol Park, Dongyoon Han, Seungryul BaekarXiv 2025 · DOI · arXiv

Introduction

The Reasoning-based Pose Estimation (RPE) benchmark serves as a critical standard for evaluating Multimodal Large Language Models (MLLMs) in their ability to interpret human poses through natural language. However, recent audits have revealed significant reliability and reproducibility flaws. This paper provides a "sanity check" by re-aligning the benchmark with its parent dataset (3DPW) and releasing refined ground-truth annotations to establish a robust foundation for future pose-aware AI.

Problem Statement: Reliability and Reproducibility Flaws

Traditional human pose estimation (HPE) focuses on geometric joint localization. RPE extends this by requiring models to identify individuals based on text descriptions before estimating their 3D pose. Despite its adoption, the RPE benchmark suffers from several systemic issues:

  1. Broken Links: Image indices in RPE often failed to match the original 3DPW dataset, necessitating manual visual matching for ground-truth verification.
  2. Data Quality and Diversity: The benchmark is characterized by redundant frames and a lack of scenario diversity, with a disproportionate number of "standing" or "sitting on stairs" poses.
  3. Linguistic Ambiguity: Text descriptions for "Shape," "Pose," and "Behavior" frequently overlap, making it difficult to determine which linguistic cues drive model performance.
  4. Incomplete Annotations: Multi-person scenes often lack comprehensive annotations, and standard square-cropping in MLLMs frequently excludes the subjects of interest.
Overview of image-level and query-level issues identified in the original RPE benchmark.

Methodology: Rigorous Data Engineering

The authors focus on benchmark auditing rather than algorithmic innovation. Their approach involved:

  • Manual Alignment: A meticulous one-to-one visual matching process to link all 50 RPE images back to high-fidelity 3DPW frames.
  • Annotation Refinement: The release of a new JSON resource containing precise SMPL (Skinned Multi-Person Linear model) parameters and 3D joint coordinates.
  • Standardized Evaluation: Re-evaluating state-of-the-art models (e.g., ChatPose, UniPose) using a consistent protocol to ensure fair comparisons.

Dataset Refinement: The Refined RPE

The primary contribution is the Refined RPE dataset. Unlike the original, it includes real_image paths, joint_cam coordinates, and smpl_param metadata, facilitating easier integration and more accurate evaluation.

Refined RPE annotation format including real image paths and SMPL parameters.

Key Insights

The Reproducibility Tax

The lack of clear mapping to source data imposed a "manual matching" tax on researchers, leading to inconsistent results and errors in metrics like Mean Per Joint Position Error (MPJPE).

The "Cropping" Blind Spot

Most MLLMs utilize center-cropping for square inputs. This often removes the target subject from the model's field of view, causing the model to "hallucinate" poses not present in the actual input.

Linguistic Overlap

Descriptions across different categories (Pose vs. Behavior) are often repetitive. For instance, "spreading hands" (Pose) and "running to the edge of the road" (Behavior) may describe the same visual state, complicating the disentanglement of linguistic reasoning.

Experimental Results

Re-evaluation of leading models using refined ground truths yielded several findings:

  • ChatPose Validation: Results aligned with original reports, validating the refinement process.
  • UniPose Discrepancy: UniPose exhibited higher MPJPE (551.3) compared to ChatPose (229.6) but performed better on Procrustes-aligned metrics (PA-MPJPE). This suggests UniPose is a strong pose generator but struggles with the linguistic reasoning required to place poses correctly in 3D space.
  • Precision Sensitivity: Inference precision (fp16 vs. bf16) significantly impacts pose accuracy, a factor previously overlooked.

Conclusion and Future Directions

While the RPE benchmark is a valuable step toward reasoning-based AI, its small scale (50 images) and scenario imbalances limit its utility. Future efforts should focus on scaling the dataset, improving multi-person annotations, and developing models that treat pose information as a supportive cue for global scene reasoning.

Summary updated

Abstract

The reasoning-based pose estimation (RPE) benchmark has emerged as a widely adopted evaluation standard for pose-aware multimodal large language models (MLLMs). Despite its significance, we identified critical reproducibility and benchmark-quality issues that hinder fair and consistent quantitative evaluations. Most notably, the benchmark utilizes different image indices from those of the original 3DPW dataset, forcing researchers into tedious and error-prone manual matching processes to obtain accurate ground-truth (GT) annotations for quantitative metrics (\eg, MPJPE, PA-MPJPE). Furthermore, our analysis reveals several inherent benchmark-quality limitations, including significant image redundancy, scenario imbalance, overly simplistic poses, and ambiguous textual descriptions, collectively undermining reliable evaluations across diverse scenarios. To alleviate manual effort and enhance reproducibility, we carefully refined the GT annotations through meticulous visual matching and publicly release these refined annotations as an open-source resource, thereby promoting consistent quantitative evaluations and facilitating future advancements in human pose-aware multimodal reasoning.
BibTeX
@misc{kim2025revisiting,
  author = {Junsu Kim and Naeun Kim and Jaeho Lee and Incheol Park and Dongyoon Han and Seungryul Baek},
  title = {{Revisiting Reliability in the Reasoning-based Pose Estimation Benchmark}},
  year = {2025},
  eprint = {2507.13314},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.13314},
}
Figure