When a pose estimator is trained on rendered people or animals and then pointed at real photographs, it often places a keypoint in the wrong spot and still reports high confidence in it. This is the "confidently wrong" failure. It is the main reason self-training on unlabeled real images tends to reinforce its own mistakes. Most practical settings where synthetic data matters, such as infant screening, clinical gait, and rare animal species, also have a small number of hand-labeled real images, often fewer than a hundred. The standard move is to fine-tune on all of them. We propose a different use for part of that budget. A fraction of the labeled real images is held out from training and used to fit a small per-joint model that predicts whether a given pseudo-label is correct. It makes this prediction from the shape of the network's own output map, from whether the prediction survives simple image transformations, and from whether the resulting skeleton is anatomically plausible.
The claim is that under a fixed budget of roughly 20 to 100 real labels, spending part of the budget on this audit beats spending all of it on gradient updates. The audit also works on isolated single frames, which the strongest existing correction mechanism cannot handle. If the claim holds, practitioners get a concrete rule for splitting a tiny label budget. If it fails, the experiments will show that plain few-shot fine-tuning from synthetic pretraining is already sufficient, and that is also worth knowing.
Background and the gap
We work with top-down pose estimators. These first detect each subject and crop a box around them, then predict keypoints inside the crop. The predictions are made by heatmap regression: the network outputs, for each joint, a grid of scores shaped like a soft bump around the likely location. The target bump is usually a 2D Gaussian, as in Flowing ConvNets. Simple Baselines showed that a ResNet with three deconvolution layers is a strong instance of this design, reaching 73.7 mAP on COCO test-dev.
Heatmaps are central to our idea. A heatmap is a whole spatial distribution, not one number per joint. It shows whether the network is sharp or diffuse, and whether a second peak competes with the first (Rectified Wing Loss discusses this multimodality). Those are exactly the signals one would want when judging whether a guess is trustworthy.
Existing synthetic-to-real pose work falls into two camps.
Unsupervised adaptation with engineered trust. These methods motivate our proposal because each documents the confidently-wrong failure and builds hand-designed machinery to work around it. None of them uses labeled real data to measure how well that machinery actually works.
- From Synthetic to Real: UDA for Animal Pose adds adversarial feature alignment, a teacher that is a running average of the student, and pruning of high-loss pseudo-labels.
- Synthetic-to-Real Pose Estimation with Geometric Reconstruction checks pseudo-labels against heatmap confidence and limb-length plausibility. It then corrects them by warping bone maps between two video frames of the same subject. This requires video pairs with a mostly static background.
- SHIFT combines a running-average teacher with a learned infant pose prior and a keypoint-to-silhouette consistency check. That check depends on segmentation masks being accurate.
Supervised use of scarce labels. PASyn shows that fine-tuning on 99 real zebra images with ImageNet initialization falls short, and that prior-aware synthetic data rescues it. It uses every real label for supervision and does not self-train. FlexPose adapts a skeleton generator from a few target poses but ignores image appearance entirely.
The gap is therefore specific. Nobody has asked whether a few real labels are worth more as judges of pseudo-labels than as teachers of the network. Nobody has built a pseudo-label filter that needs neither video nor masks.
Core Claim
A small per-joint correctness predictor is fit on held-out real labels using features of the network's own heatmaps. It can separate correct from confidently-wrong pseudo-labels much better than peak confidence can. Self-training filtered this way, starting from synthetic pretraining, should beat fine-tuning on the full label budget at the same budget.
The approach is sample-efficient for a simple reason. Every labeled image supplies one training example per joint. Fifty images of a 17-joint skeleton give about 850 correct-or-not examples. That is enough for a regularized logistic model even though it is far too few to retrain a deep network safely.
Research Program
Base model. We pretrain a top-down heatmap network (HRNet-W32 with Gaussian targets and squared-error loss) on synthetic data: SURREAL for humans, and SynAP for zebras.
Label split. Given $N$ labeled real images, we split them into a supervision set of size $N_s$ and an audit set of size $N_a = N - N_s$. The network briefly fine-tunes on the supervision set only. It is then run on the audit set.
Audit features. For each joint $j$ of each image we compute a feature vector $\phi_j$ with three parts:
- Heatmap shape: peak height, spatial spread, entropy, and the ratio of the second-highest local peak to the highest.
- Transformation consistency: how far the predicted location moves under horizontal flip, small rotation, and rescaling of the crop. This is a single-frame stand-in for the cross-frame consistency that Lin et al. obtain from video.
- Kinematic plausibility: the deviation of adjacent limb-length ratios from their range in the synthetic training poses.
Correctness predictor. A per-joint logistic model estimates
$$p_j = \sigma\left(w_j^\top \phi_j + b_j\right),$$
where $p_j$ is the predicted probability that joint $j$'s prediction lies within the PCK threshold, $\sigma$ is the logistic function, and $w_j, b_j$ are fit on the audit set by cross-validation with an L2 penalty.
Self-training. We self-train on unlabeled real images, keeping each pseudo-label only if $p_j$ exceeds a threshold chosen on the audit set to hit a target precision. The supervision set stays in every training batch. The audit set never enters gradient updates.
We then refit the predictor after each self-training round, because the network's heatmaps change as it learns and a stale audit would drift. The audit set stays clean because it is never trained on.
Analysis. Three questions drive the analysis. First, the curve of final accuracy against the split fraction $N_a/N$ for each $N$. Second, how well $p_j$ separates correct from incorrect pseudo-labels (area under the ROC curve, and precision at matched recall) compared with raw peak height. Third, which feature group carries the signal, per joint type, with particular attention to distal joints such as wrists and ankles, which PASyn reports are hardest.
Baselines
Do nothing. The naive reference is the synthetic-pretrained network applied zero-shot to real test images. On zebras this is strong: PASyn reports 90.2% [email protected] on Zebra-300 with SynAP alone.
ImageNet fine-tuning. HRNet-W32 initialized from ImageNet and fine-tuned on all $N$ real labels. PASyn reports 78.7% [email protected] at $N=99$.
Synthetic pretraining plus full fine-tuning. This is the strongest competitor, because it is exactly what a practitioner would do and it wastes no labels. PASyn reports 92.4% [email protected] on Zebra-300 with 99 real images plus SynAP. We run it with the same backbone, the same synthetic checkpoint, the same augmentation, and the same number of gradient steps as our full pipeline. Early stopping and learning rate are chosen by cross-validation on the labeled images, under a matched tuning budget.
Self-training with a tuned global confidence threshold. This is the key ablation. It uses the same label split and the same self-training loop, but replaces the correctness predictor with a single peak-height threshold chosen on the audit set. It isolates whether the multi-feature predictor matters, or whether simply having labeled data to set a threshold is enough.
Supervised self-training with all labels. A running-average teacher trained with all $N$ labels in the supervised loss, with no audit set. It isolates the effect of withholding labels from training.
Unsupervised adaptation. We compare to the unsupervised adaptation of Li and Lee on TigDog, where they report 79.50 [email protected] on horses. On infants we compare to SHIFT, which reports 56.40% [email protected] for SURREAL→MINI-RGBD. These show how much a handful of labels adds beyond purely unsupervised methods.
What counts as winning. On humans and infants, beating synthetic-plus-full-fine-tuning by at least 3 PCK points at $N \le 50$, consistently across five random label draws. On zebras, whose ceiling is near 92.4, the win condition is matching that figure with $N=30$ rather than 99. Any gain must also hold against the tuned-threshold ablation. Otherwise the contribution reduces to "use labels to pick a threshold."
Why not just…?
Why not just fine-tune the synthetic-pretrained model on all the real labels? This is the real threat. On zebras it already gets within a few points of what looks like the ceiling, according to PASyn. Our bet is that the headroom lies in domains with larger appearance gaps. The infant zero-shot figure of 56.40% in SHIFT suggests the gap there is large, and fine-tuning on 30–50 images cannot close it without unlabeled data. The split-fraction curve settles this directly: if the best split is always $N_a = 0$, the proposal is wrong.
Why not just self-train with a confidence threshold set on the labeled data? Lin et al. and Li and Lee both report that peak confidence is unreliable after synthetic training. A tuned scalar threshold may still recover much of the benefit, which is why it is our central ablation. If the ROC curves show that peak height matches the full feature set, we will report that the simpler filter suffices.
Why not replace heatmaps with 1D coordinate classification, as in SimCC? SimCC is cheaper and wins at low resolution, gaining 11.2 AP at 64×64 on COCO. Its per-axis distributions also expose peak height and spread. We will run the same pipeline with a SimCC head. The comparison tests whether the 2D shape features, especially the second-peak ratio, carry information that the 1D factorization loses.
Why not use Lin et al.'s geometric reconstruction? It needs video pairs of the same subject with a static background. Our target cohorts are single frames. Where Human3.6M video is available, we will run their method as an upper reference.
Proposed Experiments
Datasets. SURREAL→Human3.6M and 3DPW (2D keypoints, PCK); SURREAL→SyRIP and MINI-RGBD for infants ([email protected]); SynAP→Zebra-300 and Zoo Zebra ([email protected]). For each, $N \in \{10, 20, 50, 100\}$ with five random draws.
Controls. Ground-truth crops are used throughout, so that detector errors, the bottleneck discussed in Rethinking pose estimation in crowds, do not confound the comparison. We add one detector-crop run as a sanity check.
Decisive checks. The first decisive check is the split-fraction curve: does it peak at $N_a > 0$? The second is whether the predictor's ROC area on held-out real joints clearly exceeds that of peak height. The third is whether self-training driven by the predictor beats both full fine-tuning and the tuned-threshold ablation, with gains concentrated on wrists and ankles.
Secondary study. We swap the Gaussian-MSE target for the transport-based loss of Heatmap Distribution Matching, which reports +1.5 AP for HRNet-W32 on COCO val in the fully supervised setting. The question is whether its tighter heatmaps make the shape features more or less informative. Since that paper's loss requires 1,000 Sinkhorn iterations per training step, we treat this as a secondary study.
References
- Flowing ConvNets for Human Pose Estimation in Videos
- Simple Baselines for Human Pose Estimation and Tracking
- Rectified Wing Loss for Efficient and Robust Facial Landmark Localisation with Convolutional Neural Networks
- From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation
- Synthetic-to-Real Pose Estimation with Geometric Reconstruction
- Leveraging Synthetic Adult Datasets for Unsupervised Infant Pose Estimation
- Prior-Aware Synthetic Data to the Rescue: Animal Pose Estimation with Very Limited Real Data
- FlexPose: Pose Distribution Adaptation with Few-shot Guidance
- Learning from Synthetic Humans
- Is 2D Heatmap Representation Even Necessary for Human Pose Estimation?
- Rethinking pose estimation in crowds: overcoming the detection information bottleneck and ambiguity
- Heatmap Distribution Matching for Human Pose Estimation