Pose estimators for animals, hands and bodies are often trained on synthetic renders, because labeling real images is slow. The synthetic images differ from real ones in two ways. The first is appearance: texture, lighting and sensor noise. The second is geometry: the synthetic model's limb lengths, resting poses and typical postures do not match those of real subjects. Unpaired image translation, which learns to map one image collection onto another without matched pairs, handles appearance well. It handles geometry badly. A generator trained to fool a discriminator will stretch, delete or invent thin limbs, and each such edit silently invalidates the keypoint labels that made the synthetic data useful. On fruit flies, pose estimators trained on CycleGAN-translated images reach only 72.9% correct keypoints, against 84.7% when the geometric change is modeled explicitly with a 2D warp (Deformation-Aware Unpaired Image Translation for Pose Estimation on Laboratory Animals).
We propose to model the geometric change in the space of skeleton parameters rather than pixels. A small network reads the synthetic model's joint angles and bone lengths and outputs corrections. A fixed, differentiable forward kinematics layer (the chain of rotations that turns joint angles and bone lengths into 3D joint positions) converts the corrected parameters into a skeleton. A differentiable silhouette renderer projects that skeleton, and a discriminator compares the projection against real silhouettes. Only after the geometry matches the real distribution does a one-sided appearance translator paint the image. Because every generated image comes from an explicit skeleton, its keypoint labels are exact by construction, including under out-of-plane rotation and self-occlusion, where a 2D warp field has no meaningful answer.
Core Claim
A single translation network cannot both match a target distribution and preserve labels when source and target geometry genuinely differ. Preserving labels forbids geometric change, and matching the target requires it. The 2D-warping approach of Li et al. resolves this by making the change explicit, but in image space. It warps silhouettes with spatial transformers and vector fields, then carries keypoints along the warp. Its authors' own limitation is that a 2D field cannot represent a leg rotating behind the body. Our claim is that the same decomposition, carried out in kinematic parameter space, keeps the label-exactness benefit and extends it to 3D articulation. On flies, the gains should appear on the keypoints that are occluded or rotated out of plane, where the 2D warp is weakest. Anatomical plausibility comes for free, since bone lengths stay rigid within a subject and joint limits can be clamped.
Relation to Prior Work
The closest work, and the one supplying the core mechanism, is Neural Kinematic Networks for Unsupervised Motion Retargetting. It places a differentiable forward kinematics layer inside an adversarial, cycle-consistent translator between characters with different bone lengths, and reaches a joint MSE of 7.10 on Mixamo against 13.65 for a conditional RNN. It never touches pixels. Its discriminator sees motion capture, not images. We keep its central move, adversarial matching of kinematic parameters through forward kinematics, and replace its supervision signal with a silhouette discriminator on unlabeled real images. That substitution is what turns motion retargeting into sim-to-real image translation.
Two papers provide the rendering path. Mask as Supervision renders bones as differentiable Gaussian line segments and adds a small network that predicts body thickness so the rendering can match silhouettes; we reuse that renderer. It also documents that silhouettes cannot tell left from right, which constrains our design. We do not claim that silhouette adversaries resolve left–right ambiguity. We only require that the corrected parameter distribution match. The labels stay exact because they come from the source skeleton, never from inference on real images.
Skeleton2Mesh supplies evidence that amortized differentiable kinematics beats iterative fitting on cost: 0.019 s against 30.6 s per iteration. That motivates learning a correction network instead of fitting each render offline.
For appearance, Contrastive Learning for Unpaired Image-to-Image Translation replaces the round-trip reconstruction of CycleGAN with a patch-level contrastive loss. That loss asks each output patch to resemble its corresponding input patch more than other patches. It uses one generator and reports an FID of 45.5 against CycleGAN's 77.2 on Horse→Zebra. We use it as the appearance stage because a rendered part map and a real photograph carry very unequal information, which is exactly the setting where a round-trip constraint breaks.
GANerated Hands adds a geometric consistency loss to CycleGAN so that translation preserves hand structure, and fits a kinematic hand model only afterwards. That design freezes geometry, which is the right choice when synthetic and real geometry already agree and the wrong one when they do not.
Unpaired Pose Guided Human Image Generation translates fixed pose sketches into photographs. It likewise never adjusts the pose itself.
The earlier results in Deep Kinematic Pose Regression and Compositional Human Pose Regression show that predicting through a kinematic structure is more stable than predicting free joint coordinates or raw angles. This motivates correcting parameters through forward kinematics rather than moving free 3D points.
Research Program
The source is a rigged synthetic model, such as the fly model used by Li et al. or SMPL-based Surreal bodies. Each sample has joint angles $\theta$, bone lengths $b$ and a camera. A correction network $K$ outputs per-subject bone scale factors $s$ and per-joint rotation offsets $\Delta\theta$. It is conditioned on the source parameters and a noise vector, so a single synthetic pose can map to several plausible real ones.
The forward kinematics layer $f$ produces joints $x = f(\theta+\Delta\theta,\, s\odot b)$. The Gaussian line-segment renderer $R$, together with a thickness network, produces a silhouette $R(x)$. The corrections are trained with
$$\mathcal{L}_K = \mathcal{L}_{\text{adv}}\big(R(x),\, M_{\text{real}}\big) + \lambda_1 \|\Delta\theta\|^2 + \lambda_2 \|\log s\|^2 + \lambda_3\, \mathcal{L}_{\text{limits}}(\theta+\Delta\theta),$$
where $M_{\text{real}}$ are silhouettes of real images, $\mathcal{L}_{\text{adv}}$ is a standard adversarial loss, the two quadratic terms keep corrections minimal, and $\mathcal{L}_{\text{limits}}$ penalizes angles outside anatomical ranges. The real silhouettes come from background subtraction for laboratory animals filmed on uniform backgrounds, and from an off-the-shelf segmenter for humans.
The corrected skeleton is rendered as a part-labeled structure map, meaning each limb segment is drawn in its own color. A contrastive-loss appearance generator translates that map into a real-looking image. Its keypoint labels are the forward kinematics outputs $x$ projected through the camera. A standard pose estimator is then trained on these images and evaluated on real, labeled test images it never saw during translation.
Three analyses accompany the headline number. The first is per-joint accuracy split by occlusion and depth rotation. The second compares the distribution of learned bone scales against held-out real measurements, as a sanity check that $K$ learns anatomy rather than exploiting the discriminator. The third is a failure audit of images where silhouette ambiguity plausibly produced a mirrored correction.
Baselines
The naive reference point trains the same pose estimator directly on untranslated synthetic renders. A second naive reference translates those renders with CUT alone, with no geometric stage. Both show how much the geometry gap, as opposed to the appearance gap, costs.
The strongest existing method is the deformation-aware translator of Li et al.. It reports 84.7% PCK@15px on fruit flies (percentage of keypoints within 15 pixels of the truth), against 72.9% for CycleGAN and 88.8% for an estimator trained on manually labeled real data. It also reports 90.3% against 45.3% PCK@5px on worms. It is the right competitor because it attacks exactly our problem, a geometry gap with labels preserved, using a 2D mechanism. We will run it with the authors' protocol, the same synthetic source, the same unlabeled real images and the same downstream pose network. Its training steps and hyperparameter search budget will match ours. Beating it means a statistically reliable gain in mean PCK@15px on flies over several seeds. More importantly, it means a clear gain on the occluded and out-of-plane keypoint subset. A tie on the full set together with a win on that subset would still support the claim. A loss on that subset would refute it.
A CUT translator with an added keypoint-consistency loss, in the style of GeoConGAN from GANerated Hands, represents the "freeze geometry" design. It receives the same training steps and tuning budget.
The key ablation replaces forward kinematics with free per-joint 3D offsets that pass through the same renderer and adversary, with no bone-length rigidity and no hierarchy. This isolates whether kinematic structure, and not merely 3D parameterization, is responsible for any gain. A second ablation removes the joint-limit term.
For humans, going from Surreal to Human3.6M, we found no reported result on Lacuna for unsupervised synthetic-to-real translation with a pose estimator trained on the output. The unsupervised mask-supervised figure of 95.9 mm MPJPE (mean per-joint position error) from Mask as Supervision is only a loose contextual reference, because its setting differs.
Why not just…?
Why not just randomize bone lengths and poses widely in the synthetic renderer, then translate appearance with CUT? This is the most serious threat. Broad randomization is cheap and also keeps labels exact. Its weakness is that it spreads the training distribution rather than matching the target, so model capacity is spent on implausible anatomy. We test it directly. We sweep randomization width, matched in dataset size, against the learned corrections. If the best randomization width matches our method on PCK, including the occluded subset, then the learned kinematic stage is not worth its complexity, and we will report that outcome.
Why not just add a keypoint-consistency loss to CycleGAN or CUT? Such a loss preserves the source geometry. That is correct only if the source geometry is already right. When synthetic flies have the wrong leg proportions, a translator constrained to preserve them either fails to match the target or quietly violates the constraint. The 72.9% against 84.7% gap between CycleGAN and explicit deformation in Li et al. indicates that geometry change matters. The keypoint-consistency baseline above measures how much of that gap a consistency loss alone recovers.
Why not just fit the kinematic model to each real silhouette offline and use the fits as pseudo-labels? Per-image fitting is slow, as the timing comparison in Skeleton2Mesh shows. It also inherits silhouette ambiguity directly into the labels, since a mirrored or depth-flipped fit becomes a wrong label. Our method uses silhouettes only to match distributions, and its labels come from known skeletons. We will run offline fitting as a baseline with a matched compute budget. Label errors on the fitted real images can be measured against the held-out annotations.
Why not just use the 2D deformation-aware method, which already works? On largely planar views it may well be sufficient. The proposal stands or falls on whether 3D articulation matters in the target data, which is why the decisive metric is the occlusion and out-of-plane subset rather than the overall average.
Proposed Experiments
The primary experiment runs on the laboratory-animal benchmark of Li et al., using fruit flies with PCK@15px. We will also attempt zebrafish, for which the file reports no number. Worms are deliberately excluded from the main claim. A worm body is closer to a continuous curve than a jointed tree, so it tests whether a kinematic chain is the wrong prior there; a loss to 2D warping on worms is an expected and informative result.
The secondary experiment translates Surreal to Human3.6M and 3DPW, reporting MPJPE and Procrustes-aligned MPJPE of a fixed pose network trained on the generated data. A hand variant translates synthetic hands to FreiHAND (FreiHAND), reporting mesh and joint error.
The decisive checks are three. First, on flies, the method must beat the 2D warp on occluded and out-of-plane keypoints. Second, forward kinematics must beat the free 3D offset ablation, which shows that kinematic structure matters. Third, learned corrections must beat the best tuned randomization, which shows that learning the target distribution matters. Image-quality FID is reported but is secondary to downstream pose accuracy.
References
- Deformation-Aware Unpaired Image Translation for Pose Estimation on Laboratory Animals
- Neural Kinematic Networks for Unsupervised Motion Retargetting
- Mask as Supervision: Leveraging Unified Mask Information for Unsupervised 3D Pose Estimation
- Skeleton2Mesh: Kinematics Prior Injected Unsupervised Human Mesh Recovery
- Contrastive Learning for Unpaired Image-to-Image Translation
- GANerated Hands for Real-Time 3D Hand Tracking From Monocular RGB
- Unpaired Pose Guided Human Image Generation
- Deep Kinematic Pose Regression
- Compositional Human Pose Regression
- FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images