Does the Face Decide the Hair? Counterfactual Auditing of Single-Image 3D Hair Reconstruction

Single-image 3D hair reconstruction recovers a full head of individual hair strands from one portrait. Recent systems condition on features of the whole image, face included. The face is useful because it tells the model where the scalp is and helps it guess what the hidden back of the head looks like. In training data, though, the face is also statistically tied to hair texture. Public 3D strand collections are dominated by straight and wavy styles, and tight curls and afro textures are rare. A model can therefore learn a shortcut: read the face, infer which hair the training data usually paired with that kind of face, and reconstruct that hair. When it takes this shortcut it ignores the hair actually visible in the photo. A person with tightly coiled hair whose face resembles the straight-hair majority would be reconstructed with smoothed, loosened curls. In the fairness literature this is called proxy variable bias: an input that is not itself sensitive (here, facial appearance) carries demographic information and quietly steers the output.

This proposal builds a measurement instrument for that failure. The instrument rests on one fact that tabular fairness research lacks: in a synthetic rendering pipeline the counterfactual is directly observable. We can render the same 3D hair asset on many different heads and faces under the same camera and lights. Any change in the reconstructed hair is then caused by the face alone. We use these exact counterfactual pairs to measure how much and in which direction the face pulls the reconstructed hair, separately for the coarse hairstyle shape and the fine curl detail, and separately for scalp regions the camera sees and regions it does not. We then test whether simple fixes remove the effect without degrading reconstruction of the hidden back of the head.

Core Claim

Our central claim is that face-driven proxy influence in 3D hair reconstruction can be measured exactly rather than inferred through an assumed causal graph. We further claim that the right fairness criterion is not full invariance to the face but evidence dominance. Where hair is visible, the reconstruction should be fully determined by the visible hair, and changing the face should change nothing. Where hair is hidden, some use of the face as a prior may be legitimate, but its size and direction should be measured and reported. Full invariance would be the wrong target because the face carries real information about scalp position, and possibly about occluded geometry. Treating all face dependence as bias would throw that information away. Evidence dominance isolates the harmful case: the model overriding what it can see.

This sidesteps the main weakness of counterfactual fairness as formalized in Counterfactual Fairness. That paper defines fairness as a prediction that does not change under an intervention on the protected attribute, and computes the intervention by inferring hidden background variables under a hand-specified causal graph. Its guarantees depend on that graph being correct, and the counterfactual can never be observed for a real person. In a renderer we set the hair and the face ourselves, so the intervention is a real one. The causal question narrows to the well-posed question of how the reconstruction responds to the face when the hair is held fixed.

Relation to Prior Work

The closest prior work is Counterfactual Generation and Fairness Evaluation Using Adversarially Learned Inference. It generates counterfactual 2D CelebA faces by editing attributes in a learned latent space, and shows that an "attractive" classifier's output shifts sharply when skin is edited to pale. It then reduces a bias score for the black-hair-plus-pale-skin combination from 0.159 to 0.032 with a regularizer that penalizes prediction changes between factual and counterfactual images. We reuse the logic of a consistency penalty between factual and counterfactual inputs. We differ in three ways. Our counterfactuals come from rendering rather than from a learned image editor whose edits may themselves be biased. Our output is 3D strand geometry rather than a binary label. And we treat hair as the protected outcome rather than as a categorical colour attribute.

Three graphics papers motivate the gap. 3DGH: 3D Head Generation with Composable Hair and Face builds a 3D head generator with separate face and hair branches. The authors found that fully independent branches looked incoherent, so they added cross-attention from face to hair features. They explicitly flag that their study of hair–face correlation covers only gender and that broader ethnic correlations remain unstudied. DiffLocks: Generating 3D Hair from a Single Image using Diffusion Models is the system we audit. It trains a diffusion model, a generator that learns to turn random noise into samples step by step, to produce a 256×256 map over the scalp whose cells hold 64-dimensional strand codes. The diffusion model is conditioned on features from a frozen DINOv2 image encoder. DiffLocks also built a 40,000-asset synthetic dataset that deliberately includes afro and tightly curled hair. That makes it the strongest available system on textured hair, and its procedural Blender pipeline is what makes counterfactual re-rendering possible. Its whole-image conditioning has never been examined for face-driven influence.

Perm: A Parametric Representation for Multi-Style 3D Hair Modeling supplies our measuring coordinates. It represents each strand by a Fourier transform followed by a linear principal-component model: 10 coefficients describe the smooth "guide" shape and 54 describe the curl detail. It reports a strand position error of 0.020 on USC-HairSalon, against 1.211 for a frequency-domain variational autoencoder. We use Perm's split to ask whether face influence lives in coarse shape or in curl. Perm itself does no fairness analysis, and we do not use it as a generator.

On the fairness side, Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting shows that scrubbing explicit gender words from biographies still leaves gender recoverable at over 65% accuracy. That result constrains our design: masking the face is unlikely to be sufficient, because skin at the hairline, ears, neck and head shape remain visible. Mitigating Unwanted Biases with Adversarial Learning supplies an adversarial debiasing baseline, in which a second network tries to recover the protected attribute and the main model is trained to defeat it. On UCI Adult it moves accuracy from 86.0% to 84.5% while nearly equalizing false positive rates. An end-to-end fairness framework based on counterfactual reasoning: auditing, diagnosis, and mitigation separates two questions: whether a feature could act as a proxy, and whether the model actually uses it. We borrow that two-stage distinction, but that work is built for tabular data and cannot localize leakage in a spatial scalp map.

One terminology warning applies throughout. In the hair literature "proxy" also means a geometric stand-in such as a guide mesh or a visual hull. Here it always means a non-sensitive input that carries demographic information.

Research Program

The counterfactual render set. We take hair assets from the DiffLocks synthetic set, stratified by curl frequency (measured by the energy in Perm's 54 curl coefficients) and by length. We render each asset on a panel of heads sampled from a statistical head model in the style of SPHEAR: Spherical Head Registration for Complete Statistical 3D Modeling, varying face shape and skin albedo. The scalp attachment is held fixed after registration, and camera and lighting are held fixed as well. Each render comes with a per-scalp-cell visibility mask computed from the renderer, which yields a "visible" and an "occluded" region for each view. A second, smaller set varies only skin albedo with geometry fixed, which separates pigment effects from morphology.

Measuring sensitivity without demographic labels. For a hair asset $h$ rendered with faces $f$ and $f'$, let $\hat c(f,h)$ be the Perm coefficients obtained by projecting DiffLocks' reconstructed strands. The coefficients are restricted either to the guide band or to the curl band, and either to visible or to occluded scalp cells.

Sensitivity is the average change $S = \mathbb{E}\,\lVert \hat c(f,h) - \hat c(f',h)\rVert$. We report it relative to ordinary reconstruction error, so that a sensitivity comparable to the error counts as large.

Sensitivity alone cannot tell stereotyped pull from noise. We therefore also measure proxy pull. On the training set we fit a regressor $r(f)$ that predicts typical hair coefficients from face features alone; it captures what the training data associates with a face. Proxy pull is the correlation between the change in prediction, $\hat c(f,h)-\hat c(f',h)$, and the change in that association, $r(f)-r(f')$.

Positive pull means the reconstruction moves toward the hair the training data expected for that face. This defines the bias relative to the training distribution and needs no racial labels for the measurement itself. Faces are additionally grouped by skin albedo and morphology clusters only to report who is affected.

Separating proxy bias from capacity limits. The file records an open dispute. Perm attributes failure on afro and braided hair to missing data, while DiffLocks argues that linear guide-based models cannot represent such hair at all. The render set splits curl error into two parts. The part that persists for every face is a face-independent floor reflecting representation capacity or data coverage. The part that varies with the face, and correlates with $r$, is the proxy component. Reporting both tells practitioners whether to fix the data, the representation, or the conditioning.

Mitigation. We fine-tune DiffLocks with an added consistency loss on counterfactual pairs. The loss penalizes differences in predicted scalp codes between the two renders, weighted by the visibility mask. Visible cells get full weight; occluded cells get a tunable weight $\lambda$, swept from 0 to 1. This trades the fairness criterion against the use of a legitimate prior, much as the plausibility weight in the end-to-end tabular framework trades fairness against utility. The ordinary reconstruction loss is kept throughout.

Baselines

The naive reference point is DiffLocks as released, evaluated on the same counterfactual render set. Its scores define the problem's size: if its visible-region sensitivity and proxy pull are near zero, the research question is answered negatively and that is itself the result. DiffLocks is also the strongest existing method for curly and afro hair. The file states only that it beats HairStep and NeuralHDHair on precision and F-score, so there is no reported result found on Lacuna for DiffLocks precision or F-score. We will therefore recompute its strand precision, recall and F-score ourselves on held-out synthetic assets and on the Yuksel et al. benchmark used in DiffLocks' own evaluation. Those recomputed numbers are the bar that any mitigated model must not fall meaningfully below.

The blinding baseline masks the face and fills it with a neutral skin texture before encoding, keeping the hairline and head outline so the scalp can still be located. It is run on the same renders with the same fine-tuning budget as our method.

The decorrelation baseline retrains DiffLocks with face–hair pairings randomly shuffled during data generation. This breaks the training-set association between face and hair.

The adversarial baseline follows Mitigating Unwanted Biases with Adversarial Learning. It attaches an adversary to the conditioning tokens that tries to predict $r(f)$, and removes the corresponding gradient component from the main model's updates.

All mitigated models share the same base checkpoint, the same synthetic data, an equal number of fine-tuning steps, and an equal hyperparameter search budget. Each must hold strand F-score within the seed-to-seed noise of the base model.

The key ablation compares consistency loss on all regions ($\lambda=1$) with visible-only consistency ($\lambda=0$). This isolates evidence dominance as the ingredient that matters. Our method beats a baseline if it drives visible-region proxy pull to statistical zero while retaining lower occluded-region error.

For calibration of how large a fairness cost is usually tolerated, the cited tabular results are the only anchors in the file. Counterfactual fairness raised RMSE from 0.873 to 0.918–0.929 on law school data, and adversarial debiasing cost 1.5 accuracy points on Adult. A comparable relative cost in strand F-score would be acceptable.

Why not just…?

Why not just crop or mask the face? Masking cannot remove skin at the hairline, the ears, the neck, or overall head shape, and Bias in Bios shows that scrubbing explicit cues leaves the protected signal largely recoverable from what remains. Masking also distorts the input distribution the encoder was trained on. The blinding baseline tests this directly. If it achieves near-zero proxy pull at equal F-score, we will report that the cheap fix suffices for this pipeline.

Why not just shuffle face–hair pairings in the synthetic training data? This is the real threat, and we think it may well work on synthetic test data, since it removes the correlation the shortcut feeds on. Its cost is that it also removes any legitimate face prior for the hidden back of the head, and it does nothing for models that must train on real captures where pairings cannot be shuffled. The experiment settles this in two parts. We compare occluded-region error under decorrelation against the $\lambda$ sweep. We also test on real portraits, described below, whether decorrelation's gains survive the synthetic-to-real gap. If decorrelation matches our method on both, the contribution becomes the audit instrument together with the recommendation to decorrelate, and we will say so.

Why not just debias the representation adversarially? Adversarial training removes information about the protected attribute from the representation everywhere, including the scalp-location information the face legitimately provides. It is also known to be unstable. It is run as a baseline. We expect it either to leave residual pull or to cost F-score, and the comparison at a matched F-score will show which.

Proposed Experiments and Decisive Checks

The first decisive check is whether the problem exists. That requires DiffLocks' visible-region proxy pull to be significantly positive and its curl-band sensitivity to be large relative to reconstruction error, with confidence intervals bootstrapped over hair assets. The second check is localization: whether pull concentrates in the curl band (the fine-texture smoothing story) or the guide band (stereotyped overall style), and in visible or occluded regions. The third check is the face-independent floor versus proxy decomposition of curl error on tightly coiled assets, which bears directly on the dispute between Perm and DiffLocks. The fourth check is the mitigation comparison at matched F-score.

For real images, we select K-Hairstyle portraits (K-Hairstyle) with annotated curl strength and multi-view captures. The set is restricted to Korean subjects, which limits generality, and we state that limit plainly. On this set we compare curl statistics of each reconstruction re-rendered into held-out views against orientation statistics extracted from the real held-out views. We also test whether the reconstruction error correlates with $r(f)$ more for the base model than for mitigated models. A clear negative outcome, in which DiffLocks shows no visible-region pull, would still be useful: it would give the community a reusable counterfactual audit and a measured floor showing where curl failure comes from instead.