Many of the best models for 3D human pose work at several scales at once. They run computations on the full skeleton, then merge joints into coarser groups such as "both ankles" or "left arm", run computations on those groups, and pass the results back down. The groupings are always chosen by hand from anatomical intuition. No one has checked whether a particular grouping helps, or whether any merge of roughly the right size would do equally well. This note proposes to answer that by treating the grouping, called the pooling schedule, as the one variable in a controlled population of models. We would train hundreds of otherwise-identical 3D pose-lifting networks, each with a different schedule, and use tools from architecture interpretability to attribute accuracy to specific joint merges.
The claim to test is concrete. Most of the benefit of multi-scale skeletal pooling should come from a small number of structural properties of the schedule: whether merged joints are neighbors along a kinematic chain, how much the joint count shrinks at each level, and whether opposite-side limbs are merged. The specific anatomical labels should matter much less. If the claim holds, hand-designed hierarchies can be replaced by short, checkable rules. If it fails, and the hand-designed groupings beat every rule-based alternative, the field gains the first quantitative evidence that anatomical intuition is doing real work.
Core Claim and What Is New
Hand-designed hierarchies are now standard. Graph Stacked Hourglass Networks (GraphSH) merges 16 joints into 8 and then 4 by averaging left–right pairs, and reaches 35.8 mm mean per-joint position error (MPJPE) on Human3.6M from ground-truth 2D joints. MS-TGN uses fixed 25/11/7-joint subsets. ST-PGN averages joints into limb groups. GUESS cascades through 22/11/5/1 nodes. These papers motivate the work: each shows that hierarchy helps, and none tests its own grouping against alternatives.
GraphSH's left–right merge is a clear suspect. It merges the two ankles even in walking, where the legs move in opposite phase. Merging two signals that are often opposite in sign may destroy information. That is a testable mechanism, not just an aesthetic complaint.
The closest bridging work, Towards Part-Aware Monocular 3D Human Pose Estimation, differs in three ways. It searches convolution cells over image feature maps for body parts fixed in advance, and it shows specialization only by swapping architectures between parts. We do the reverse on all three counts: the part partition is the variable, the network is a skeletal graph model, and the explanation comes from formal attribution rather than swapping.
MS-TGN reports that generic architecture search produced bloated skeleton models. That finding constrains our design. We will not search over operations at all, only over the pooling schedule, inside an otherwise fixed and well-tuned network.
Research Program
The design space. We would reimplement GraphSH and keep everything fixed except the two pooling matrices (16→8 and 8→4) and their matching unpooling matrices. These include the operations, channel width of 64, four stacked hourglasses, attention fusion, and training recipe.
We would sample schedules from several sub-spaces:
- the original left–right pairing;
- kinematic-chain merges, where merged joints form connected pieces of the skeleton tree (for example ankle with knee, or wrist with elbow);
- arbitrary partitions with the same group sizes;
- partitions that deliberately merge distant, unconnected joints;
- schedules with different shrink ratios, for example 16→10→4 or 16→6→2.
This follows the method of Designing Network Design Spaces. That paper compares whole populations of models through error empirical distribution functions under a short training budget, rather than crowning a single winner. We use it because individual-model gaps in this literature are about 2 mm, which is close to seed noise.
Attribution. We encode each schedule as a small labeled tree: joints are leaves, clusters are internal nodes, and labels record joint identity and chain adjacency. We then fit the Weisfeiler–Lehman-kernel Gaussian-process surrogate from NAS-BOWL. That surrogate compares graphs by counting shared small subtrees. NAS-BOWL showed that taking the derivative of its predictions with respect to each subtree feature gives a per-motif contribution to accuracy. Here a motif is a specific merge, such as "left ankle with right ankle at level 1" or "ankle with knee at level 1". The surrogate's cubic cost in the number of models is not a problem, because our population is in the hundreds.
As a check that does not depend on the surrogate, we adapt Operation Importance from On Redundancy and Diversity in Cell-based NAS. In trained models, we split one merged cluster back into its joints at test time and measure the rise in error.
Representation checks. We would compute linear centered kernel alignment (CKA), a similarity score between the feature representations of two layers, across levels and across the four stacked hourglasses. Refining Remote Photoplethysmography Architectures using CKA used this to show that half of a 10-layer 3D-CNN was redundant. Here it tests whether coarse levels carry information distinct from fine levels, and whether later hourglasses are redundant under good versus bad schedules.
Rules and transfer. Following the PrimSkip logic, we distill the top-ranked motifs into one or two rules, for example "merge only chain-adjacent joints, never opposite-side limbs below the torso". We then test whether random schedules obeying the rules match the best searched schedule.
Baselines
Flat model (do-nothing reference). The weakest reference is a flat graph network with no pooling, trained under the same recipe. Prior GCNs reach 37.8 mm MPJPE on Human3.6M with ground-truth 2D input, as reported in GraphSH. We rerun our own flat version so that differences in recipe do not contaminate the comparison.
GraphSH with its hand-designed schedule (strongest existing method). This is the strongest competitor and the one the question is about: 35.8 mm MPJPE, 3.70M parameters, from ground-truth 2D joints. We retrain it with five seeds under both our short screening budget and the full published schedule. Every alternative schedule receives exactly the same epochs, optimizer, hyperparameter search budget, and seeds. Beating it means one of two things: a rule-derived schedule whose seed-averaged full-schedule MPJPE falls below GraphSH's by more than twice the seed standard deviation, or matching it while also reducing error on the opposite-phase action subsets described below.
SasMamba reports 41.48 mm P1 with 0.64M parameters. The file does not state whether that uses ground-truth or detected 2D input, so it serves only as context, not as a direct comparison.
Learned soft pooling. DiffPool learns cluster assignments end to end. We drop it into the same network with the same cluster counts and tuning budget.
Random partitions of matched size (key ablation). This ablation isolates the core claim. If arbitrary partitions with GraphSH's group sizes perform as well as anatomical ones, then the shrink ratio is doing the work, not the grouping.
Why not just…?
Why not just let the network learn the pooling with DiffPool? This is the real threat, and it is a baseline for that reason. On a 16-node graph, learned soft assignments may well match hand-designed ones in accuracy. However, soft assignments that change with the input do not, on their own, say which merges matter. Pooling Architecture Search and DiffPool were built for molecules and citation graphs and ignore kinematic structure. If DiffPool wins outright, the program pivots: we would apply the same attribution and CKA analysis to its learned assignments and ask whether it rediscovers chain-adjacent merges. The answer would be equally informative.
Why not just drop hierarchy and use a flat attention or state-space model? The evidence in the file runs against this. SasMamba found that flattening skeletons hurt until multi-scale strided scanning was added. GUESS's coarse-to-fine cascade cut FID from 0.473 for the flat model MLD to 0.109. The flat-model baseline above tests this directly in our setting.
Why not just tune GraphSH's schedule by hand or by plain random search? The population is in fact a random search. What hand-tuning or random search alone cannot provide is an explanation that transfers. A best-of-N schedule may simply be a lucky seed. The motif attribution, the rule distillation, and the transfer tests below are what separate a finding from a lucky draw.
Decisive Experiments
The primary screen runs Human3.6M lifting from ground-truth 2D joints, Protocol 1 and Protocol 2. It uses roughly 400 schedules at a short budget, then retrains the top and bottom 10 per sub-space with five seeds at full budget. Before any comparison, we would check that rank order is preserved between the short and full budgets, using Spearman correlation.
Test 1 (the core claim). We compare the error distributions of the kinematic-chain, left–right, arbitrary, and distant-merge sub-spaces. The claim predicts that chain-merge schedules dominate and that arbitrary schedules approach GraphSH. Both outcomes are publishable. If anatomical left–right pairing clearly dominates every alternative, the claim is falsified.
Test 2 (the opposite-phase mechanism). We break down error by action, comparing Walking with Sitting and other actions where the legs move together. We also run the merge-splitting attribution on the ankle pair. The prediction is that splitting the left–right ankle merge lowers error specifically on Walking.
Test 3 (transfer). The rule-derived schedules, trained on Human3.6M, are tested on MPI-INF-3DHP. GraphSH reports no MPI-INF-3DHP number in the file, so we measure both models ourselves. The same rules are also applied to MS-TGN-style action recognition on NTU RGB+D, cross-subject split, against MS-TGN's reported 89.5%.
Test 4 (representation checks). CKA should show that good schedules keep coarse levels distinct from fine levels, while poor schedules produce collapsed or redundant later hourglasses.
References
- Graph Stacked Hourglass Networks for 3D Human Pose Estimation
- A Novel Graph Representation for Skeleton-based Action Recognition
- Spatio-Temporal Pyramid Graph Convolutions for Human Action Recognition and Postural Assessment
- GUESS: GradUally Enriching SyntheSis for Text-Driven Human Motion Generation
- Towards Part-Aware Monocular 3D Human Pose Estimation: An Architecture Search Approach
- Designing Network Design Spaces
- Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
- On Redundancy and Diversity in Cell-based Neural Architecture Search
- Refining Remote Photoplethysmography Architectures using CKA and Empirical Methods
- SasMamba: A Lightweight Structure-Aware Stride State Space Model for 3D Human Pose Estimation
- Hierarchical Graph Representation Learning with Differentiable Pooling
- Pooling Architecture Search for Graph Classification