Human pose estimators, the models that recover a 3D body from an image, do not agree on what a skeleton is. One model outputs the 24-joint tree of the SMPL body model. Another outputs the 17 joints of the Human3.6M motion-capture setup. A third lifts COCO's 17 surface keypoints into 3D. Each convention places "the hip" or "the shoulder" at a different physical point. Today these models are compared by keeping only the joints they share, or by pushing everything through a linear "joint regressor" (a fixed matrix that turns mesh vertices into joint positions). Then mean per-joint position error (MPJPE, in millimetres) is averaged over the body. Natural-language parsing faced the same situation two decades ago: parsers built on incompatible grammar theories could not be scored against each other. It learned that converting one format into another hides conversion errors inside the score. It also learned that averaged metrics can reverse rankings.
We propose to carry the parsing community's best answer, conversion-free evaluation on targeted phenomena, into pose estimation. The measuring instrument is a differentiable forward-kinematics layer: a fixed function that turns joint rotations into 3D joint positions along a bone tree. Each model's native output is fitted by one shared kinematic tree, with a small calibrated offset table for each skeleton convention. Models are then scored only on joint angles every convention can express, such as knee flexion, elbow flexion and shoulder abduction. Rotations that a convention cannot express, such as forearm twist from sparse keypoints, are not counted against it. The fit residual separately reports when the instrument, rather than the model, has failed. The central claim is testable: if one body is written out in different skeleton conventions, this metric should give near-identical scores, while intersection-MPJPE and regressor-based comparison should not.
What prior work established and what it leaves open
On the parsing side, three results motivate the design. Parser Evaluation Across Frameworks without Format Conversion scores parsers by running small recognizers on each parser's native output to detect specific constructions. Its argument is that a conversion script with its own error rate makes model failure and conversion failure indistinguishable. We adopt the native-recognizer idea, and we also make recognizer failure measurable, which Tam et al. could only assert (their recognizers succeeded 100% on a 10-sentence suite). Cross-Framework Evaluation for Statistical Parsing maps both constituency and dependency trees onto a consensus tree and gives zero cost to edits that only one theory can make. We borrow that rule for degrees of freedom that a skeleton convention lacks. Treebank Annotation Schemes and Parser Evaluation for German shows the danger of aggregate scores. The same 85 inserted attachment errors cost 1.16 points on the flat TIGER treebank but only 0.43 on the denser TüBa-D/Z, and a dependency metric reversed the ranking. We reuse its controlled error-insertion design as our main validation.
On the kinematics side, Deep Kinematic Pose Regression supplies the mechanism. Its layer predicts angles, chains rigid transforms down a fixed tree, and computes loss on positions. Its comparison against regressing angles directly explains why our recognizer fits positions rather than angles: angle-space errors are ill-conditioned along a chain. Neural Kinematic Networks for Unsupervised Motion Retargetting shows that an FK layer conditioned on target bone lengths can move motion between bodies. It assumes identical topology, and that assumption is exactly what we drop. HybrIK splits rotations into a closed-form swing plus a learned twist. IKOL solves inverse kinematics as a differentiable Gauss–Newton layer. Both convert sparse joints into SMPL's tree, and IKOL is our strongest conversion competitor. A Simple Method to Boost Human Pose Estimation Accuracy by Correcting the Joint Regressor for the Human3.6m Dataset is the clearest evidence that the yardstick itself is biased. Refitting the Human3.6M regressor lowers SPIN's MPJPE from 60.4 to 56.9 mm without retraining, so some of the reported error is measurement artifact. Their fix is one recalibrated matrix for one dataset; we want a protocol that spans conventions. AdaptPose improves cross-dataset accuracy but evaluates on harmonized joint intersections, the practice we test. Compositional Human Pose Regression reports that bones vary far less than absolute joint positions. That supports reading pose off bone directions and angles rather than joint positions.
Core Claim
A pose metric defined on shared joint angles, measured through a calibrated consensus FK layer, is invariant to skeleton convention in a way that position metrics are not. It also attributes disagreements correctly between model error and measurement error. Concretely, the recognizer for convention $f$ estimates consensus-tree rotations $\hat\theta$ from a model's native joints $x_j$:
$$\hat\theta = \arg\min_{\theta} \sum_{j \in f} \big\| \mathrm{FK}_f(\theta;\, b^f)_j - x_j \big\|^2 + \lambda R(\theta),$$
Here $j$ ranges over the joints of convention $f$, and $\mathrm{FK}_f$ chains rigid transforms down the consensus tree. It emits the points that convention $f$ calls joints, placed using a per-convention offset table $b^f$. $R$ is a weak joint-limit prior with weight $\lambda$. The phenomenon score is the angular error between $\hat\theta$ and gold on the subset of rotations identifiable from convention $f$. The minimized residual is reported separately as recognizer failure. Identifiability is decided from the rank of the FK Jacobian for that convention, not chosen by hand. This is the kinematic version of zeroing theory-specific edits.
Research Program
We first build the consensus tree as the generalization of the SMPL, Human3.6M and COCO graphs. It contains every joint that at least two conventions locate, and records which conventions observe which rotations.
Next we calibrate the offsets $b^f$ without image data. We take ground-truth SMPL meshes from 3DPW and the corrected pseudo-ground-truth Human3.6M meshes from Hedlin et al. We emit each convention's joints from the same mesh, using SMPL's joint regressor, the standard and corrected Human3.6M regressors, and a COCO surface-keypoint regressor, and we fit $b^f$ jointly with $\hat\theta$. Calibration and test subjects are disjoint.
We then define phenomena as named shared rotations: knee and elbow flexion, hip and shoulder abduction/flexion, spine bend, and, where identifiable, forearm and thigh twist. Gold values come from 3DPW's IMU-fitted SMPL and from Human3.6M motion capture, both read through the same recognizer.
Finally, we re-rank published models across conventions, using released checkpoints. These include SPIN, VIBE, HybrIK and IKOL (SMPL tree, via the IKOL code and HybrIK loaders), and kinematic or compositional Human3.6M-skeleton regressors. For each, we report the per-phenomenon angle error, the recognizer residual, and the coverage rate. Coverage is the fraction of frames whose residual exceeds a threshold, which mirrors the roughly 15% no-parse rate of the deep grammar in Ivanova et al.. We use it to measure where rigid-skeleton outputs break under occlusion and domain shift on MPI-INF-3DHP.
Baselines
The naive reference is each model's native MPJPE and PA-MPJPE (MPJPE after rigid alignment) on its own skeleton. These numbers are what papers report today. The 3DPW PA-MPJPE values in the file are IKOL 44.5 mm, HybrIK 48.8 mm and SPIN 59.2 mm, from IKOL. We will show where native rankings disagree with phenomenon rankings.
The second baseline is intersection-joint MPJPE, the AdaptPose-style practice of keeping only identically named joints. It runs on exactly the same frames and outputs as our metric.
The third baseline converts every output into Human3.6M joints through the corrected regressor of Hedlin et al.. It is the best existing calibration-based fix; its reported gains are SPIN 60.4→56.9 mm, VIBE 62.9→60.0 mm and MEVA 101.6→98.3 mm MPJPE on Human3.6M. For keypoint-only models it is only applicable after a mesh fit, which we perform with the same budget as our recognizer.
The strongest competitor converts every model's joints into SMPL rotations using IKOL's Gauss–Newton layer and compares SMPL angles directly. It is strongest because it is also angle-based, uses a learned twist, and is the most accurate published inverse kinematics in the file. We run it with released weights, the same input joints, and the same per-frame compute ceiling as our recognizer. No reported result was found on Lacuna for angle-level, cross-convention scores from any of these methods, so all baseline phenomenon numbers will be produced here.
Two ablations isolate the key ingredients. The first removes the identifiability mask, scoring all consensus rotations including twist for keypoint formalisms. The second removes the per-convention offsets, using one shared $b$. Beating the baselines means two things. First, a lower formalism-invariance gap, defined below, than all four, with a clear margin over the IKOL conversion. Second, equal sensitivity to inserted errors across conventions where MPJPE-based metrics differ.
Why not just…?
Why not just report per-joint MPJPE instead of the average? This removes dilution across joints, but not the definitional mismatch. A Human3.6M "hip" and a COCO "hip" are different points, and Hedlin et al. show that even within one dataset the joint centres are systematically displaced. Per-joint error therefore still mixes convention offset with model error. The invariance test below measures this directly: per-joint MPJPE should give a nonzero gap between conventions for identical bodies.
Why not just convert everything to SMPL with IKOL or HybrIK and compare angles? This is the real threat, because it is nearly our method with a stronger learned prior. Its weakness is that it predicts twist for formalisms that never observe it. It also offers no separate signal for conversion failure, which is exactly Tam et al.'s objection. If IKOL-conversion matches our invariance gap and error-insertion sensitivity, we will say so and adopt it as the recognizer. The protocol's contribution would then be the identifiability masking, the residual-based attribution and the calibration test, not the solver.
Why not just recalibrate one regressor, as Hedlin et al. did? A single linear map fixes one target convention and cannot express rotations. Its errors also remain inside the score. We include it as a baseline so the comparison is empirical rather than argued.
Proposed Experiments
The decisive check is formalism invariance. Using the same ground-truth meshes, we generate each convention's output and score each one against gold with every metric. A theory-neutral metric should report almost zero error and an almost zero spread between conventions. The spread is the invariance gap. If our gap is not clearly below that of IKOL-conversion and intersection-MPJPE, the core claim fails.
The second check follows Rehbein and van Genabith's error insertion. We perturb a single rotation, for example knee flexion by 5, 10 or 20 degrees, in the gold mesh and re-emit it in each convention. We then measure each metric's response. Our metric should respond equally across conventions and should scale with the inserted angle. We expect MPJPE-based metrics to respond less on denser skeletons, as PARSEVAL did on deeper trees.
The third check is real re-ranking on 3DPW, Human3.6M and MPI-INF-3DHP. We report Kendall rank agreement between the native and phenomenon orderings, and list which phenomena drive any reversal.
Finally, we validate attribution. Frames with high recognizer residual should contain injected conversion faults, such as swapped joint labels or wrong offsets, and not model errors. We will measure how well the residual separates the two kinds of failure.
References
- Parser Evaluation Across Frameworks without Format Conversion
- Cross-Framework Evaluation for Statistical Parsing
- Treebank Annotation Schemes and Parser Evaluation for German
- Deep Kinematic Pose Regression
- Neural Kinematic Networks for Unsupervised Motion Retargetting
- HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation
- IKOL: Inverse Kinematics Optimization Layer for 3D Human Pose and Shape Estimation via Gauss-Newton Differentiation
- A Simple Method to Boost Human Pose Estimation Accuracy by Correcting the Joint Regressor for the Human3.6m Dataset
- AdaptPose: Cross-Dataset Adaptation for 3D Human Pose Estimation by Learnable Motion Generation
- Compositional Human Pose Regression
- On Different Approaches to Syntactic Analysis Into Bi-Lexical Dependencies. An Empirical Comparison of Direct, PCFG-Based, and HPSG-Based Parsers