Where to Put Global Context in a Video Pose Lifter When 3D Labels Are Scarce

Turning a video of a person into a 3D skeleton usually takes two steps. A 2D detector first finds joint positions in each frame. A "lifter" network then converts the sequence of 2D joints into 3D. Current lifters build their depth by repeating one block: the same attention layer, the same state-space scan, or the same dilated temporal convolution, stacked many times. The same design rule appears in their temporal handling. Each block either mixes information across the whole body and many frames, or across a small neighbourhood. That choice is fixed and does not change from the first layer to the last.

Work in language modelling and image classification suggests that this uniformity wastes capacity. When architecture search is free to choose, it tends to put local operators early and global operators late, or global operators only at the ends of the network. This note asks whether that depth-wise ordering matters for 3D pose from video, and in particular where it should matter most: when only a few thousand labelled 3D frames exist. The bet is that choosing which layers see local context and which see global context is a form of prior knowledge. Prior knowledge pays off most when data is scarce, and it should show up as smoother motion, not only as lower per-frame error.

Core Claim

The proposal makes three claims. First, a temporal pose lifter whose operators change with depth can match uniform lifters at the same parameter count and compute. Its early layers would be short-window temporal convolutions that act within limb groups. Its late layers would be full-body, long-window attention. Under full supervision it should roughly tie with uniform lifters. Second, when 3D labels are restricted to a single Human3.6M subject or a few thousand frames, it should clearly beat them on both joint error and acceleration error. Third, the advantage should shrink but not vanish when every architecture first receives the same self-supervised pretraining on unlabelled 2D motion.

The new element is not "a hybrid convolution-attention lifter." The proposal is a controlled study that crosses four factors: the depth-wise placement of local versus global operators, the amount of labelled 3D data, the presence of pretraining, and the resulting temporal smoothness. Its aim is to find out whether architectural ordering can stand in for unlabelled data, and by how much.

Why This Gap Is Real

Every leading lifter in the file is uniform across depth. PoseFormer stacks identical spatial and temporal Transformer modules over up to 81 frames. MotionBERT stacks identical dual-stream blocks, each running spatial-first and temporal-first attention in parallel. PoseMamba is the closest relative. It mixes local and global context inside each block by interleaving a limb-grouped scan with a full-body scan, but it repeats that same block at every depth. Pavllo et al. repeat dilated temporal convolutions uniformly.

The only lifter in the file that is heterogeneous across stages is GLA-GCN. It runs a global graph-aggregation stage followed by per-joint unshared output layers, and it uses the opposite ordering from the one hypothesised here: global first, local last. That paper does not contradict this proposal, because its "local" stage is a per-joint decoder rather than a stage of local context. It does show that the ordering question is open, and the experiments below test its order directly against ours. Its design was also hand-built, and it was trained only with full supervision.

The case for depth-wise ordering comes from other domains. In language, LV-BERT searched over layer sequences and found that dynamic convolutions belong early and self-attention late. That small model reached 78.8 on GLUE against 77.0 for ELECTRA-small. STAR evolved language-model layouts and found something different: attention at the first and last layers and gated convolutions in the middle. That is a second candidate ordering we must test. In image classification, TResNet uses simple blocks early and bottleneck blocks late. Together these papers motivate the study. They also constrain it, because they disagree on the right ordering, so we cannot assume one.

The link to label scarcity comes from The Heterogeneity Hypothesis. Its layer-differentiated channel widths gave higher training error but lower test error, which is the signature of a regulariser. One example is MobileNetV1 on Tiny-ImageNet, where Top-1 error fell from 51.87% to 46.44% with 37% of the parameters. That result concerns widths rather than operator types, and it does not involve pose. We separate the two effects with an ablation.

Research Program

We will build one family of lifters from a shared pool of operators. The local operator is a short-window temporal convolution, applied separately within five limb groups and the torso, following the anatomical grouping that PoseMamba uses for its local scan. The global operator is spatial attention over all joints combined with temporal attention over the full input window, as in PoseFormer. Each model has the same number of layers and hidden width. Parameters and multiply-accumulate operations are matched to within five percent by adjusting the convolution kernel width.

Five depth layouts are compared:

  • all-global, essentially a MotionBERT-style uniform stack;
  • all-local, essentially a grouped temporal convolutional network;
  • local-to-global, our hypothesis, with roughly the first two thirds local;
  • global-to-local, the GLA-GCN ordering;
  • a sandwich with global layers at both ends and local layers in the middle, the STAR ordering.

A sixth variant keeps the all-global operators but applies single-shot channel shrinkage from The Heterogeneity Hypothesis. This separates "any heterogeneity regularises" from "local-before-global ordering specifically helps."

Before training any of these, we will run a cheap diagnostic on a trained uniform model. For each layer, we will measure how far across joints and frames its attention reaches on average. If early layers in a fully trained uniform lifter already attend mostly within a limb and within a few frames, that supports local-to-global. If not, it predicts that the ordering will not matter, and we report that outcome.

Each layout is then trained under a grid of label budgets on Human3.6M, crossed with two initialisations. The label budgets are: subject S1 only; a 5,000-frame subset matching the low-resource setting of Pavllo et al.; subjects S1 and S5; and the full training split. The initialisations are random weights, or pretraining by recovering 3D motion from masked and noisy 2D skeletons as in MotionBERT. Pretraining uses the same unlabelled data and the same number of steps for every layout, and a masking ratio below the 45% point where MotionBERT reports its representations breaking down.

Inputs are detected 2D keypoints throughout, with ground-truth 2D used as a secondary condition so that results can be compared with GLA-GCN's reported numbers. We report three metrics on Human3.6M:

  • MPJPE, the mean per-joint position error in millimetres;
  • P-MPJPE, the same error after rigid alignment;
  • acceleration error in millimetres per frame squared, which measures jitter.

Cross-dataset transfer uses MPI-INF-3DHP and HumanEva-I without fine-tuning. Fit3D and AIST, which QuaMo uses for fast motions, are included for acceleration error if the data can be obtained. No model receives any post-hoc smoothing.

Baselines

The naive reference point is a per-frame lifter: the all-global layout with a temporal window of one frame, followed by a Gaussian filter whose width is tuned on validation data. It fixes the floor for how much temporal context buys at all. The tuned filter is also a fair stand-in for "just smooth it."

The strongest existing method is MotionBERT. It reports 37.5mm MPJPE on Human3.6M, and it is the only competitor in the file that already combines pretraining with a spatio-temporal lifter. That makes it the one that most directly threatens the claim that architecture matters once pretraining is present. We will retrain it under our label budgets with its published recipe, the same pretraining data, and the same tuning budget of twenty learning-rate and weight-decay trials per label budget, using the same budget for every model.

PoseMamba reports 38.1mm MPJPE with 16% of MotionBERT's compute. It is the strongest efficiency competitor and the closest design, because it mixes local and global scans within uniform blocks. It will be run at matched compute.

Pavllo et al. is the strongest label-efficiency competitor. Its semi-supervised back-projection improved accuracy by 10 to 15mm with 5,000 labelled frames, but no absolute low-label MPJPE was found on Lacuna for this setting. We will therefore reproduce it, using the camera intrinsics it requires, rather than quote a figure. For jitter, the reference is QuaMo's reported acceleration error of 5.3 mm/frame² on Human3.6M.

The key ablations are the all-global layout, the global-to-local and sandwich layouts, and the width-shrinkage variant. Together they show whether gains come from ordering, from heterogeneity in general, or from neither.

Beating the baselines means the following. Under full supervision, local-to-global should be within 1mm MPJPE of matched-compute MotionBERT and PoseMamba. In the S1-only and 5,000-frame regimes without pretraining, it should beat all-global and both alternative orderings by at least 3mm MPJPE and show lower acceleration error. That gap should hold across three seeds. With pretraining, it should either keep a smaller but consistent gap, or match pretrained MotionBERT at lower compute.

Why not just…?

Why not just pretrain a uniform lifter and fine-tune it? This is the most serious threat. MotionBERT and, in radiography, Masked Record Modeling show that self-supervised pretraining can bring 1–10% label regimes close to full supervision. If pretraining erases the gap between layouts, then the architectural prior is redundant whenever unlabelled 2D motion is available. The full crossing of layout and pretraining is designed to measure exactly this. A vanishing gap is a valid and reportable outcome. In that case the remaining contribution would be lower compute at equal accuracy, and we would state it that way.

Why not just smooth the output of a strong per-frame lifter? Filters treat all frames alike. As Rempe et al. and QuaMo document, uniform smoothing produces foot sliding during stance and lag during fast transitions. The naive baseline tests this directly by tuning the filter generously. The proposal only wins if it reaches lower acceleration error without paying in MPJPE on Fit3D- and AIST-style fast motion.

Why not just use Pavllo's temporal convolutions with back-projection? That approach is cheap and label-efficient. However, it needs camera intrinsics, and its uniform blocks pool all joints together, which GLA-GCN argues causes interference between body parts. Our all-local layout is close to this baseline. If all-local matches local-to-global in the low-label regime, the global layers are unnecessary there, and we will say so.

Why not just run an architecture search, as LV-BERT and STAR did? Search would answer "which layout is best" at high compute. It would not tell us why, or whether the answer changes with label budget. The disagreement between LV-BERT's local-early result and STAR's attention-at-the-ends result shows that searched layouts are domain-specific. Five hand-chosen layouts that span the hypotheses cost a small fraction of a search and give an interpretable answer. If the ordering effect proves real, a targeted search is the natural follow-up.

Decisive Checks

The single most informative result is the gap in MPJPE and acceleration error between local-to-global and all-global under S1-only labels. It must be measured at matched compute and reported with and without pretraining. If the gap is near zero without pretraining, the hypothesis is wrong. If it is large without pretraining and zero with it, architectural ordering substitutes for unlabelled data but does not add to it, which is useful to know before building larger lifters. If it survives pretraining, depth-wise ordering is a lasting design rule for video pose.

Two secondary checks complete the picture. The first is whether local-to-global beats global-to-local and the sandwich layout, which settles the ordering conflict between GLA-GCN, LV-BERT and STAR in this domain. The second is whether the width-shrinkage variant recovers the same benefit, which would point to heterogeneity in general rather than ordering. The per-layer attention-reach diagnostic, run on the uniform model, should predict the winning ordering in advance. If it does not, that disagreement is itself a reportable finding.