Score-Field Audits of Synthetic Minority Faces

Teams that fix demographic imbalance in face recognition increasingly do it with generated images. They synthesize faces of under-represented groups, train on them, and report success on benchmarks. Two questions are rarely checked. Where does the synthetic data differ from real faces of the same group? And does that difference predict which groups will actually improve when the model is tested on real photographs? Today the first question is answered by human inspection, and the second only after a full training run. This proposal builds an automated, label-free check that answers both before any training happens.

The check estimates the score function, meaning the gradient of the log-probability density, $\nabla \log p(x)$. This is a vector field that points, at each location, toward where data becomes more likely. We estimate one field for real face embeddings and one for synthetic face embeddings, and compare them region by region. We use a family of estimators that fit the gradient directly in closed form instead of fitting a density and then differentiating it. The comparison yields two things: a per-sample filter, and a per-region forecast of real-world fairness gains. The forecast is then tested against real benchmarks, so the audit itself does not rest on simulation alone.

Core Claim

Our central claim is that, inside the embedding space of a face recognizer, the mismatch between the real and synthetic score fields locates where a synthetic cohort fails. The two failures we target are missing within-identity variation and drift in texture or lighting. This mismatch also predicts, without demographic or identity labels on the real data, how much each demographic group's real-world verification accuracy will change when that synthetic data is added. A filter built on the mismatch should beat Chameleon-style one-class filtering at a matched number of retained samples.

The quantity we compute is the local Fisher divergence restricted to a region $R$ of embedding space:

$$D_R = \mathbb{E}_{x \sim p_{\text{syn}},\, x \in R}\, \big\| g_{\text{real}}(x) - g_{\text{syn}}(x) \big\|^2 .$$

Here $g_{\text{real}}$ and $g_{\text{syn}}$ are the estimated score fields of real and synthetic embeddings, and the regions $R$ are found by mode-seeking clustering on the real score field, with no labels.

One extra statistic targets the intra-class failure that the FRCSyn 2024 challenge reports. For a roughly Gaussian identity cluster with covariance $\Sigma$, the average squared score norm equals the trace of the inverse covariance:

$$\mathbb{E}\|\nabla \log p(x)\|^2 = \operatorname{tr}(\Sigma^{-1}).$$

Synthetic identities with too little pose and lighting variation therefore show inflated score norms relative to real identities. This gives a measurable, directional signature of "not enough intra-class variation."

Where This Sits

The method comes from the least-squares log-density gradient line of work. In that line, the score-matching objective is minimized over models that are linear in Gaussian kernel features, which gives an analytic solution. It was introduced for mode-seeking clustering in Clustering via Mode Seeking by Direct Estimation of the Gradient of a Log-Density. It was extended by Regularized Multitask Learning for Multidimensional Log-Density Gradient Estimation, whose multi-task variant (MT-LSLDG) shares strength across coordinates. That variant reached an adjusted Rand index of 0.48 on 36-dimensional satellite data, where mean shift built on a kernel density estimate scored 0.00. We reuse these estimators unchanged as the measurement device. Their prior evaluations never touched face embeddings, demographic skew, or synthetic-versus-real comparisons.

One constraint matters in this setting. Recognizers trained with margin losses produce embeddings normalized onto a sphere. The integration-by-parts step behind score matching assumes the density vanishes at an unbounded boundary, and a sphere has no such boundary in the ambient coordinates. We therefore use the manifold formulation of Least-Squares Log-Density Gradient Clustering for Riemannian Manifolds, and compare it against a tangent-space projection.

The motivating evidence comes from synthetic-fairness work. Chameleon generates minority faces by in-painting with DALL-E 2. It found that automated quality metrics missed generation artifacts, so it fell back on a one-class SVM plus human screening. Our filter is a drop-in replacement for that screening step. FRCSyn 2024 shows that diffusion-generated cohorts can shrink demographic gaps but lack intra-class variation. That finding is the specific failure our score-norm statistic targets.

Two findings constrain the claim. Meta Balanced Network shows that ArcFace trained on perfectly balanced real data still drops from about 92% on the lightest skin tone to about 83% on the darkest. Better data alone therefore cannot close the gap, and we claim only that our audit predicts and improves the data contribution. Racial Faces in the Wild points to sensor-level contrast problems on darker skin. This is exactly the kind of real-world structured noise that clean synthetic renders omit, and our regional audit should flag it.

Research Program

We freeze a face encoder: ArcFace trained on CASIA-WebFace, plus a second copy trained on BUPT-Balancedface. We map three sets of images into its embedding space. The first is an unlabeled real reference sample drawn from BUPT-Globalface. The second is a set of synthetic cohorts produced with the generator families used in FRCSyn, namely identity-conditioned Stable Diffusion with PhotoMaker and IDiff-Face. The third is Chameleon-style in-painted minority faces.

We then reduce the embeddings to tens of dimensions on the sphere, or project them to a tangent PCA. We fit MT-LSLDG with a few thousand kernel centers, using block coordinate descent where exact solves are too costly. Mode-seeking on the real field defines the regions. For each region we compute $D_R$, and for each synthetic identity cluster we compute the score-norm ratio relative to real identity clusters.

The audit is used in two ways. As a forecast, we correlate regional $D_R$, mapped to demographic groups only after the fact, with the measured change in per-group real accuracy after augmentation. As a filter, we drop the synthetic samples with the largest pointwise discrepancy, $\|g_{\text{real}}(x) - g_{\text{syn}}(x)\|$.

We also test whether the audit is itself biased. An encoder trained on Caucasian-heavy data compresses minority faces into a small area of embedding space. That compression could hide synthetic errors for exactly those groups. Comparing audits from the CASIA and BUPT-Balancedface encoders tells us whether this happens.

Baselines

The naive reference is to add all synthetic samples with no filtering, alongside a real-only model, both trained with identical recipes. The second reference is random filtering at the same retained count, which separates the effect of filtering from the effect of having less data.

The strongest existing filtering method is Chameleon's one-class SVM rejection, minus the human step, run on the same embeddings with the same retention budget. Chameleon's reported result is on a different task: F1-disparity for the Black group in race prediction on FERETDB fell from 79% to 27% (Chameleon). We will rerun it on our verification setup instead of comparing to that number.

The strongest fairness-at-training references are IMAN, which raised RFW African-face accuracy from 85.13% to 91.42% with an ArcFace backbone (Racial Faces in the Wild), and MBN, which lowered the Skewed Error Ratio on Identity Shades from 2.33 to 1.59 (Meta Balanced Network). We report these as context, not as targets: they change the training objective, whereas we change the data. For synthetic augmentation, the context figures are FRCSyn's 95.42% overall accuracy for real-plus-synthetic training and its −5.31% GAP for bias mitigation (FRCSyn 2024).

For the forecasting question, which is the paper's main claim, there is no reported result on Lacuna for predicting per-group real-world gain from synthetic data. Our predictors will instead be compared against per-group FID-style feature distances and against classifier-based scores.

Two ablations isolate the key ingredient. The first replaces the score difference with the log density ratio value alone, which tests whether gradient information matters. The second swaps MT-LSLDG for single-task LSLDG, and the Riemannian estimator for a Euclidean one. Every method gets the same encoder, the same retention counts, the same downstream training compute, and the same hyperparameter search budget, tuned on a held-out real split separate from the evaluation sets.

Beating the baselines means two things. The score filter must give a lower Identity Shades Skewed Error Ratio and higher RFW minimum-group accuracy than one-class filtering at equal retention, across three seeds. The rank correlation between $D_R$ and per-group gain must clearly exceed that of the FID-style predictor.

Why not just…?

Why not just differentiate a kernel density estimate? MT-LSLDG reports that differentiating a kernel density estimate amplifies noise and fractures structure in moderate dimensions: ARI of 0.00 versus 0.48 on satellite data in Regularized Multitask Learning for Multidimensional Log-Density Gradient Estimation. We still include it as a baseline at our working dimension, because face embeddings after projection may be easier than those benchmarks.

Why not just train a real-versus-synthetic classifier and use its output? This is the real threat. A well-calibrated classifier estimates the log density ratio, and the gradient of that ratio is exactly our score difference. If the classifier's input-gradient matches our forecast and filter performance, then the direct estimator adds only closed-form convenience. We would report that outcome honestly and reframe the contribution around the intra-class score-norm diagnostic, which a classifier does not provide directly. The classifier and its input-gradient are both included at matched capacity and tuning.

Why not just compute FID per demographic group? Per-group FID compares global means and covariances. It requires demographic labels on the real data and gives no location for the failure. It serves as our main forecasting baseline. If it predicts per-group gains as well as $D_R$, our regional claim fails.

Proposed Experiments

The decisive check is the forecast. We train with each synthetic cohort added to CASIA-WebFace and measure per-group accuracy change on RFW and per-tone accuracy on Identity Shades. We then ask whether audit-time $D_R$, computed without labels, ranks those changes better than per-group FID and the classifier score.

The second check is filtering at retention rates of 25%, 50%, and 75%. It reports Identity Shades Skewed Error Ratio, RFW minimum-group accuracy, and AgeDB and CFP-FP accuracy, to confirm that filtering does not trade overall accuracy for fairness.

The third check tests whether the identity score-norm ratio tracks real intra-class failure, measured as verification accuracy on pose-varied CFP-FP and the hard partition of GBU, across generators.

The fourth check repeats the audit with the balanced-trained encoder. If minority-group discrepancies rise sharply under that encoder, the encoder's compression of minority faces was hiding synthetic errors. That would be a finding in its own right, and it would argue for always auditing in a balanced-trained encoder's embedding space.