Skin Tone or Lesion Contrast? Nonlinear Conditional Audits of Dermatology AI

Dermatology image classifiers work worse on darker skin. The usual way to check this is to sort test images into the six Fitzpatrick skin-type bins and compare accuracy across bins. Fixes then target "skin tone": rebalance the data, or train the network so its features carry no skin-type information. Two findings in the literature suggest this framing points at the wrong variable and uses the wrong measuring tool. DDI-CoCo annotated biopsy-confirmed images with the color contrast between lesion and surrounding healthy skin. On high-contrast images, the gap between light and dark skin nearly vanished across CNN and transformer backbones. That suggests much of the "skin tone gap" may run through low local contrast, which happens more often on pigmented skin. Separately, work on nonlinear bias leakage shows that zero linear correlation between a model's output and a sensitive attribute can coexist with strong nonlinear dependence, such as a U-shaped one. Grari et al. describe such U-shaped dependence, and FairGroups shows that coarse bins can hide large disparities inside a bin.

We propose to measure where dermatology classifiers' errors actually depend on continuous optical properties of the image, using a nonlinear conditional dependence test. The question is whether errors depend on skin lightness once the true diagnosis and the lesion's local contrast are held fixed. We then use the audit's answer to choose and test the training fix. If tone still matters after accounting for contrast, a nonlinear penalty conditioned on continuous tone and contrast should help where binned and linear methods do not. If tone stops mattering once contrast is accounted for, the right fix is to make the model better at low-contrast lesions, and independence penalties on tone are the wrong tool. Either outcome changes what the field should build.

Core Claim

The central quantity is the conditional dependence between a model's per-image prediction error $e$ and the image's skin lightness $t$, given the true label $y$ and the lesion-to-skin contrast $c$. We measure it with conditional distance correlation, written $\mathrm{dCor}(e, t \mid y, c)$. Distance correlation is a dependence measure that is zero only under full statistical independence, so it catches U-shaped and other non-monotonic relationships that Pearson correlation misses. We make three claims, each checked on held-out data. First, linear and Fitzpatrick-binned audits underestimate how much error depends on tone and contrast. Second, they wrongly attribute contrast-driven error to tone. Third, a training penalty that drives this conditional dependence toward zero, with continuous tone and contrast as conditioning variables, narrows the dark-skin gap beyond what the existing dermatology debiasing method achieves.

This differs from the closest work in three ways.

DISCO supplies the method. It enforces prediction independence from a bias variable given the label, and it factorizes the conditional distance correlation estimate down to quadratic memory. Its bias variables are discrete (race groups in FairFace, positions in dSprites), and it is tested on facial and synthetic tasks. We condition on a continuous, multi-dimensional optical profile in a 114-class clinical task. We also separate a nuisance we want removed (tone) from a difficulty we want the model to overcome (contrast).

FairDisCo is the dermatology competitor. It removes skin-type information from features using a shallow adversarial "confusion" head that tries to predict skin bins, plus a contrastive loss. It works on binned skin type and on the representation rather than the error. Scrubbing tone from the features is arguably the wrong goal in dermatology, because lesions genuinely look different on different skin.

FairGroups finds data-driven partitions of continuous tone and fixes scores after training. We borrow its partitions as an evaluation lens, not as the intervention.

Research Program

The first component is measuring the optical variables. Tone will be measured as $L^*$ (lightness in CIE-L*a*b* color space) and the Individual Typology Angle, taken from healthy-skin patches rather than the whole image. Groh et al. report that whole-image Individual Typology Angle is unstable under lighting, shadow and redness, and we treat that as a known constraint on the design. Contrast follows the DDI-CoCo protocol, which computes Web Content Accessibility Guidelines (WCAG) contrast from three lesion points and three adjacent healthy-skin points.

Applying this to Fitzpatrick 17k, which has no such annotations, requires an automatic estimator. We will train a lightweight point-proposal model on DDI-CoCo's annotations and validate it against a manually annotated stratified subset of about 500 Fitzpatrick 17k images. If its contrast estimates correlate poorly with the human annotations, the Fitzpatrick 17k contrast analyses are reported as exploratory only.

The second component is the audit. For a trained classifier, we estimate $\mathrm{dCor}(e, t \mid y, c)$ and the unconditional $\mathrm{dCor}(e, t \mid y)$. Here $e$ is per-image cross-entropy or the true-class margin. Conditioning on the continuous $c$ uses kernel weighting over nearby contrast values, with bandwidth chosen by held-out likelihood. Significance comes from conditional permutation tests, adapting the inverse conditional permutation idea of FairICP so that the null hypothesis keeps the label–contrast structure intact. This matters most on DDI, whose 656 images give limited statistical power, so every DDI claim will carry a permutation p-value and bootstrap interval.

The third component is the intervention. We add a penalty $\lambda \, \mathrm{dCor}(\hat{p}, t \mid y, c)$ to the classification loss on mini-batches. Here $\hat{p}$ is the model's predicted class-probability vector and $\lambda$ sets the penalty strength. This is a continuous version of equalized odds: among images with the same diagnosis and similar lesion contrast, predictions should not depend on skin lightness. We deliberately do not penalize dependence on contrast. Removing it would only flatten performance down to the low-contrast level, which is a ceiling effect rather than a fix. Contrast is instead addressed by a separate contrast-targeted training arm, so the audit's verdict can be checked against which arm actually helps.

Baselines

The naive reference point is empirical risk minimization: the same backbone trained on cross-entropy with no fairness term. The cited references are VGG-16 at 20.2% accuracy on Fitzpatrick 17k (Groh et al.), and DeepDerm on DDI at ROC-AUC 0.61 on types I–II versus 0.50 on types V–VI (Daneshjou et al.). We retrain our own version on a shared ResNet-50 and SwinV2-B so every method shares architecture, initialization, data splits, and a fixed budget of 30 hyperparameter trials.

The strongest existing dermatology method is FairDisCo. It reports +8.54% multi-class demographic disparity and +7.74% equality of opportunity improvement in-domain on Fitzpatrick 17k, and +4% AUC and +0.75% accuracy out-of-domain on DDI, relative to its baseline. We reimplement it under the same budget. Beating it means two things together: a smaller worst-partition gap on DDI types V–VI with a non-overlapping bootstrap interval, and no loss in overall accuracy.

The contrast reference is fine-tuning on DDI-CoCo with contrast-stratified sampling. DDI-CoCo reports this reaches SwinV2 AUC 0.852 on high-contrast and 0.832 on low-contrast images, with dark-skin performance matching or exceeding light-skin performance. It is the most dangerous competitor, discussed further below.

We also run GroupDRO over Fitzpatrick bins. Daneshjou et al. found GroupDRO, CORAL and CDANN failed to narrow the gap, but the file contains no per-group number for that finding, so the only result is our own rerun. We also run the original sDISCO conditioned on discrete Fitzpatrick bins; for DISCO on clinical data there is no reported result found on Lacuna.

Three ablations isolate the key ingredient. The first replaces distance correlation with a linear covariance penalty on the same continuous variables, which tests whether nonlinearity matters. The second replaces continuous tone with Fitzpatrick bins, which tests whether continuity matters. The third drops $c$ from the conditioning set, which tests whether separating contrast from tone matters.

Why not just…?

Why not just stratify the evaluation by Fitzpatrick type and contrast bins, as DDI-CoCo already does? Binned stratification is a linear, coarse audit. FairGroups shows predefined bins can hide disparity inside a bin; on CelebA its data-driven partitions captured fairness variance of 0.092 versus 0.078 for Fitzpatrick bins. The decisive test is predictive. On a held-out split, does the nonlinear conditional audit, or the binned audit, better locate the worst-performing data-driven partition of the tone-by-contrast plane? If the binned audit predicts as well, we will say the nonlinear audit is unnecessary for diagnosis.

Why not just fine-tune on contrast-annotated data and skip independence penalties? This is a real threat. DDI-CoCo fine-tuning already brought dark-skin performance to parity on DDI. Our audit makes the threat testable rather than something to argue around. If $\mathrm{dCor}(e, t \mid y, c)$ is indistinguishable from zero under permutation, the gap is contrast-mediated. In that case we expect the contrast arm to win, and we will report that the tone penalty adds nothing. The penalty earns its place only if residual tone dependence exists at fixed contrast, which is most plausible for redness-driven conditions where melanin masks color (Huang et al.). Even then, it must beat the contrast arm on that subset.

Why not just reweight or run GroupDRO on skin-type bins? Daneshjou et al. report that GroupDRO and domain adaptation did not close the types I–II versus V–VI gap. DISCO reports matching or outperforming GroupDRO on Waterbirds and similar benchmarks. Both findings come from settings other than ours, so we rerun GroupDRO under identical budgets rather than relying on them.

Proposed Experiments

We first validate the audit where the truth is known. We fine-tune a classifier on Fitzpatrick 17k after planting a U-shaped dependence: we flip labels for a chosen class at a rate that peaks at mid-range $L^*$, so linear correlation with lightness stays near zero. The nonlinear conditional audit should detect the planted dependence at the known strength. Linear and binned audits should miss it. If the nonlinear audit also misses it, the program stops there.

We then run the real audit on empirical-risk baselines trained on Fitzpatrick 17k and evaluated on Fitzpatrick 17k's held-out split, its 312-image dermatologist-verified subset, and DDI and DDI-CoCo. The key number is the ratio of conditional to unconditional tone dependence, $\mathrm{dCor}(e, t \mid y, c) / \mathrm{dCor}(e, t \mid y)$. A ratio near zero means contrast explains the tone gap. A ratio near one means it does not. We compute the ratio per condition group, separating erythema-dominant from pigmentation-dominant conditions.

Finally, we train all baselines, the proposed penalty and its three ablations on Fitzpatrick 17k, then test out-of-domain on DDI. We report overall accuracy and AUC, AUC on types V–VI, worst data-driven-partition AUC, and the residual conditional dependence.

The decisive check is a pre-registered prediction made from the audit before training. The tone penalty will outperform the contrast arm on exactly the condition groups where the audit found residual tone dependence, and not elsewhere. If the penalty helps uniformly, or helps where the audit found no dependence, the audit has failed as a diagnostic even if the penalty works, and we will report it that way.