Does the Skeleton Decide the Gender? Pose-Driven Masculine Defaults in Sign Language Video Rendering

Many systems that generate videos of people work in two steps. They first produce a stick-figure skeleton, meaning a sequence of joint positions over time, and then hand it to a renderer that paints a photorealistic person onto those joints. Sign language production and sign-video anonymization rely on this design. It is usually assumed that the skeleton is identity-free, so the person's appearance, including perceived gender, comes only from whatever appearance the system is told to use. This proposal tests that assumption directly. Skeletons are not neutral. Bone lengths, shoulder width and movement amplitude all differ between bodies, and a renderer trained on mostly one kind of body may learn to read them as gender cues. If that happens, the renderer could default to a masculine presentation when no appearance is specified. It could also override an explicitly requested feminine appearance whenever the motion came from a male body. The second case would be a visual counterpart of the "feminine penalty" that machine translation shows on sentences whose gender is unambiguous.

The work has three parts. We build a controlled swap experiment that separates how much of a rendered person's perceived gender comes from the appearance input and how much comes from the skeleton. We then split the skeleton's contribution into static body proportions and dynamic movement style. Finally, we test whether removing those channels costs sign-language intelligibility. The practical stakes are concrete. In anonymization, a skeleton that carries gender leaks a demographic attribute of the real signer. In production, it means a system cannot reliably render the signer a deaf user asks for.

Core Claim

The claim is that pose-conditioned human renderers have a second route for masculine defaults that existing audits cannot see, because it does not pass through text. Prior audits all enter through language. Beyond Content shows that grammatical masculine markers raise male depiction in text-to-image models by +51.0 percentage points while feminine markers raise female depiction by only +3.0. the Sora case study shows occupational prompts such as "CEO" producing 10/10 male videos. Both treat the generator as a black box driven by words. Sign-gloss pipelines rarely carry grammatical gender of the kind Spanish or Italian does, so the text route is a weak lens here. The skeleton is the likely carrier.

We predict two effects, stated so that either can fail. The first is an override effect: perceived gender of the output shifts toward the gender of the skeleton's source body even when a clearly feminine or clearly masculine appearance reference is supplied, and the shift is larger toward masculine than toward feminine. The second is a prior-sampling default: when appearance is sampled from the renderer's learned distribution, as in anonymization, the male fraction of outputs exceeds the male fraction of the training signers.

What Prior Work Gives Us and Leaves Open

The pose-to-video side supplies the systems we audit. AnonySIGN separates appearance from 2D OpenPose skeletons using a conditional variational-autoencoder GAN, which is a generator that learns a smooth distribution of appearances it can sample from. It renders new, non-existent signers. The authors themselves note that signing rhythm and speed survive as a biometric. They do not analyze the demographic makeup of sampled appearances, and they do not ask whether the skeleton steers appearance. That question is ours. Stable Signer blends pose priors retrieved from a database and feeds them to diffusion renderers (ControlNeXt/Wan). A diffusion model is a generator that learns to turn random noise into images step by step. This gives us a second, modern renderer family whose pose priors come from real signers of particular bodies.

A Simple Baseline … with 3D Avatars fits SMPL-X, a parametric 3D body model whose shape parameters $\beta$ control proportions separately from the pose, to signing video. We reuse its fitting with its upright-posture and temporal-smoothness losses as our tool for swapping body shape while keeping the motion. The paper evaluates only linguistic and geometric accuracy, and never asks what body shape implies about the rendered person. Behavior-Driven Synthesis of Human Dynamics separates dynamics from static posture. That motivates our further split of the skeleton channel into proportions and movement.

The bias side supplies the hypotheses and the measurement logic. Investigating Failures … Unambiguous Gender shows translation systems failing to reach 70% accuracy when explicit feminine cues contradict stereotypes. This is the pattern our override test transfers to vision, with an explicit appearance reference standing in for an explicit pronoun. A Tale of Pronouns traces masculine defaults to near-zero attribution on the source gender token. That suggests a matching diagnostic for us: does the renderer's output depend on the appearance input at all in the frames where it misrenders? Similarity-network analysis of English–Italian MT finds neutral terms drifting toward masculine anchors in embedding space. That motivates the prior-sampling test on appearance latents.

Research Program

The central object is a factorial rendering experiment. Each source clip has a skeleton from a signer of known presentation $g_p \in \{m, f\}$. Each appearance reference has presentation $g_a \in \{m, f\}$. We render every cross-pairing, and a fixed rater panel and a calibrated classifier score each output for perceived presentation. Define

$$\Delta_{\text{pose}}(g_a) = P(\hat{g}=m \mid g_p=m, g_a) - P(\hat{g}=m \mid g_p=f, g_a),$$

where $\hat{g}$ is perceived output presentation. $\Delta_{\text{pose}}$ is the share of the output's gender that is decided by the skeleton rather than the reference. A renderer that respects its appearance input has $\Delta_{\text{pose}}=0$. The override effect predicts $\Delta_{\text{pose}}>0$, and predicts it is largest for feminine references ($g_a=f$). For the prior-sampling test we report the male fraction among outputs with sampled appearance, compared against the male fraction of training signers. A ratio above one is amplification.

To separate channels, we run the same factorial on three versions of each skeleton. The first is the raw skeleton. The second is a proportion-normalized skeleton, in which we fit SMPL-X and replace $\beta$ with the appearance reference's body shape or a canonical shape, then re-project. The third is proportion-normalized with movement amplitude and speed rescaled to population means. If $\Delta_{\text{pose}}$ goes to zero at the second step, the leak is proportions alone and simple retargeting fixes it. Any residual at the second step that disappears at the third is attributable to movement style. Separately, we train a small probe that predicts source-signer gender from normalized kinematics. It gives an upper bound on how much gender the motion carries, independent of any renderer.

The mitigation is gender-adversarial training of the renderer's pose encoder: a gradient-reversal head penalizes the encoder for predicting source-signer presentation. We combine it with cross-identity pairings during training. It is only worth keeping if it beats shape-swap retargeting on $\Delta_{\text{pose}}$ without losing sign intelligibility. Intelligibility is measured by back-translation, meaning a pretrained sign-to-text model reads the rendered video and we score its BLEU-4 against the reference sentence.

Data come from RWTH-PHOENIX-Weather-2014T and How2Sign, which both have multiple signers. Signer presentation labels do not exist in either dataset. We will annotate perceived presentation per signer and report inter-annotator agreement. Motion-X (paper) serves as a non-signing control, to check whether the effects are specific to signing or general to pose renderers. The measurements are binary and concern perceived presentation, not anyone's identity. We state this limitation plainly and report an "ambiguous" rater option instead of forcing a choice.

Baselines

The naive reference point is the unmodified renderer driven by raw skeletons. We use two such renderers. One is an AnonySIGN-style cVAE-GAN retrained on our splits with the published losses. The other is a ControlNeXt-style diffusion renderer as used in Stable Signer. These establish whether $\Delta_{\text{pose}}$ and amplification exist at all. For the existing anonymization method, the file reports FID 49.48, LPIPS 0.243, a source-linkage rate of 10.5% in lineup tests, and back-translation BLEU-4 of 13.09 against 13.52 for original video on PHOENIX14T (AnonySIGN). No reported result found on Lacuna for the gender presentation of AnonySIGN outputs or for any pose-to-video renderer's $\Delta_{\text{pose}}$. Our retrained version must come within reasonable range of the published FID and BLEU before its bias numbers are trusted.

The strongest competitor to our mitigation is retargeting: SMPL-X shape-swap using the fitting from A Simple Baseline … with 3D Avatars, plus simple 2D bone-length normalization. Either can be applied at inference to any existing renderer, so they are the cheapest plausible fix. All methods share the same training data, renderer backbone, training steps and hyperparameter search budget. Each search is tuned on held-out signers and judged on unseen signers.

Beating retargeting requires two things. It means a $\Delta_{\text{pose}}$ for feminine references that is significantly lower, with confidence intervals from signer-level bootstrap. It also requires the back-translation drop relative to the raw renderer to be no larger than AnonySIGN's 0.43 BLEU-4 drop. The key ablation runs the adversarial head without cross-identity pairing, and cross-identity pairing without the adversarial head, to show which ingredient removes the kinematic residual.

Why not just…?

Why not just normalize bone lengths before rendering? This is the real threat, and the program is built so that it can win. If proportion normalization drives $\Delta_{\text{pose}}$ to zero, that is the main finding: the community should retarget by default, and the adversarial method is unnecessary. The reason to expect a residual is that AnonySIGN reports rhythm and speed surviving as identity cues after its skeleton step. Normalizing proportions does not touch those cues. The kinematic probe and the third normalization level measure exactly that residual.

Why not just supply an explicit appearance reference, since explicit prompts worked for Sora? In the Sora study, "A female CEO" did change outputs where hedged wording failed. However, the unambiguous-gender translation study shows that explicit cues are still overridden when they conflict with a strong prior. The factorial tests directly whether an explicit feminine reference survives a male-source skeleton. If $\Delta_{\text{pose}}(f)\approx 0$ for the raw renderer, explicit references suffice and we say so.

Why not just rebalance the training signers by gender? Rebalancing would shrink the prior-sampling default, and we run it as an extra condition. It need not remove the override effect, though. A renderer trained on balanced data can still learn that wide shoulders predict a masculine face, because that correlation is real in the data. A Tale of Pronouns likewise finds that broad data exposure leaves masculine defaults in instruction-tuned models. A balanced-data renderer evaluated on the same factorial shows which effect rebalancing actually fixes.

Decisive Checks

The study succeeds as a finding if three things hold. The raw renderers must show $\Delta_{\text{pose}}>0$ for feminine references, with signer-bootstrap intervals excluding zero, on both PHOENIX-2014T and How2Sign. The effect must be asymmetric, with masculine overrides larger than feminine ones, as in the translation literature. The classifier and human raters must agree on the direction. It fails cleanly if $\Delta_{\text{pose}}$ is indistinguishable from zero, which would itself support the identity-free assumption behind anonymization.

The channel split is decisive in either direction. A zero residual after shape-swap retargeting means the fix is geometric. A residual that only movement rescaling removes means motion itself carries gender, which reopens the anonymization question raised by AnonySIGN. Lineup tests of whether raters can recover source-signer gender from anonymized output, before and after mitigation, connect the result to privacy. Throughout, back-translation BLEU-4 on PHOENIX14T guards against a "fix" that removes gender by making signing unreadable.