Biosensor engineers have a standard fix for a sample that interferes with their measurement. Raw blood or saliva shifts the sensor's signal even when no target is present, so they add a reference channel that sees the same background and subtract it. This proposal applies the same fix to skin photographs. Melanin in the top layer of skin filters the light that reaches the camera, much as serum proteins and salts distort a biosensor's reading. On darker skin, this filtering is what hides redness and shrinks lesion-to-skin contrast. We propose to treat the patient's own nearby healthy skin as the reference channel. The model would classify a lesion from its optical absorbance relative to that patch, rather than from raw pixel color.
The second half of the claim is a prediction about where the skin-tone accuracy gap comes from. We expect the gap between light and dark skin to grow as the diagnostic labels become finer. Coarse distinctions such as "inflammatory vs. neoplastic" survive the loss of color contrast. Fine distinctions among 114 conditions, or among severity grades, depend on exactly the subtle color cues that melanin removes. If that holds, then much of the fairness problem in skin-image triage is a measurement problem that worsens with label granularity. A cheap physical correction, testable on a single smartphone photo, should then shrink the gap most where it is largest. The correction should also fail in a predictable way where melanin has destroyed signal rather than merely shifted it.
Core Claim
Two hypotheses can each be tested and could each be wrong. The first concerns how the gap scales with label granularity. Define the disparity at label granularity level $k$ as $\Delta(k) = \mathrm{acc}_{\text{FST I–II}}(k) - \mathrm{acc}_{\text{FST V–VI}}(k)$. Here FST is the Fitzpatrick skin type, a six-point scale from lightest (I) to darkest (VI), and $k$ indexes label sets running from a few coarse groups up to the full condition list. We predict that $\Delta(k)$ increases with $k$ for standard image classifiers.
The second hypothesis concerns what self-referencing can and cannot fix. Under a simple Beer–Lambert picture, where light is attenuated exponentially by each absorber it passes through, the camera intensity in color channel $c$ factors into three terms. These are the illumination $I_{0,c}$, a round-trip melanin transmission $T_{m,c}^2$, and the reflectance of the tissue underneath, $R_c$. The reference-subtracted absorbance is
$$\tilde{a}_c = -\log \frac{I_c^{\text{lesion}}}{I_c^{\text{ref}}} = -\log \frac{R_c^{\text{lesion}}}{R_c^{\text{ref}}},$$
which cancels both lighting and melanin to first order, provided the lesion and the reference patch share the same epidermis.
What the ratio cannot restore is the signal-to-noise ratio. When melanin attenuates strongly, the lesion's own contrast falls toward the camera's noise floor, and dividing by a reference does not bring it back. The proposal therefore predicts a residual gap after correction. We treat that residual as an estimate of information physically lost to pigment, not as a modeling failure. That floor matters for deployment, because it tells clinicians which fine-grained calls a camera-based tool should refuse to make on darker skin and route to an expert.
Where This Sits in Prior Work
The gap itself is well documented. Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology With the Fitzpatrick 17k Dataset curated 16,577 images over 114 conditions labeled by skin type. It showed that a VGG-16 model is less accurate on types V–VI, and argued that models miss the alternative appearance of inflammation on dark skin. PASSION for Dermatology collected 4,901 images from Sub-Saharan Africa focused on FST IV–VI. It also found that accuracy drops when moving between clinics in different countries. Chiu et al. pinpointed the mechanism we target: segmentation models rely on the boundary contrast between lesion and healthy skin, which baseline pigmentation distorts. These papers diagnose the problem and serve as our benchmarks. None of them separates pigmentation from lesion contrast inside the model, and none asks how the gap depends on label granularity.
Three papers constrain the design. The Fitzpatrick 17k study found that the automated skin-color score ITA (Individual Typology Angle) varies heavily with lighting and with the lesion's own redness. Conditioning a model on a global skin-tone estimate is therefore fragile. Our within-image reference ratio avoids estimating skin tone at all. Luo et al. found that larger uncurated pretraining widened subgroup gaps. That warns against assuming scale alone will absorb the correction, and it is why we test across backbone sizes. Charpignon et al. and Yfantidou et al. argue that melanin attenuation in optical sensing is a physical effect and not a software bug. We take this seriously, and it is why we predict an irreducible floor rather than full parity.
Three papers supply the method. The reference idea comes from A Self-Reference Interference Sensor Based on Coherence Multiplexing. That sensor ran a measurement chip and a reference chip through one optical path, so temperature drift and non-specific binding hit both and cancelled, cutting temperature-induced error tenfold. We replace the second chip with a patch of the patient's own healthy skin. Isolating Specific vs. Non-Specific Binding Responses in Conducting Polymer Biosensors reinforces a related lesson: interference can be separated by the form of the signal rather than removed chemically. The abstention component comes from DeepTag. DeepTag found that abstaining on the most uncertain 10% of cases sharply raised precision on fine-grained diagnoses. We make the abstention rate depend on the measured signal floor rather than on a fixed fraction.
Research Program
The system has three parts.
The first part selects a reference patch. A lightweight segmenter finds the lesion and picks an annulus of skin around it, excluding hair, specular highlights and visibly involved skin. A random distant patch serves as a control, so we can tell whether proximity matters.
The second part is a two-stream classifier. One stream receives the ordinary RGB crop. The other receives the per-channel log-ratio image from the equation above, computed against the median reference color. The streams merge before the classification head. This keeps the model free to use texture and shape, which the ratio does not change, while giving it a melanin-cancelled color signal.
The third part is granularity-aware abstention. For each skin-type band and label level $k$, we measure the lesion-to-reference contrast relative to the noise in the reference patch, a per-image signal-to-noise estimate. The model abstains on fine-grained calls when this estimate is below the level at which validation accuracy collapses. It still emits the coarser label. This mirrors the "rule-out / observe / rule-in" zones of the emergency chest-pain protocol reviewed in Performance of the 0-Hour/1-Hour Algorithm: each patient receives the finest call the evidence supports. The intended use is single-photo triage at the point of care, which is the regime where fast answers matter and expert review is scarce.
We build the granularity ladder on Fitzpatrick 17k by grouping its 114 conditions into progressively broader categories, with a dermatologist checking the groupings. That gives several levels on the same images, so $\Delta(k)$ can be traced as a curve rather than measured at one point. On PASSION, the levels are condition classification and the secondary-infection (impetigo) task.
Baselines
The naive reference point is a ResNet-50 and a VGG-16 trained on raw RGB at each granularity level with standard augmentation. The published numbers anchor it. The Fitzpatrick 17k paper reports 20.2% overall accuracy across 114 classes and 26.7% on its verified holdout. PASSION reports a ResNet-50 balanced accuracy of 0.70 for condition classification, an F1 of 0.63 for impetigo, and a 7% balanced-accuracy spread between FST IV and VI. We rerun these models ourselves, with identical splits and tuning budgets, so that every method shares one evaluation pipeline.
The strongest fairness competitor is distribution alignment in the style of FairCLIP. FairCLIP adds a loss that minimizes the Sinkhorn distance, an optimal-transport measure of how far apart two feature distributions are, between each subgroup's features and the overall population's. On Harvard-FairVLMed it reduced the race Demographic Parity Difference from 15.35% to 6.07% while raising AUC from 67.84% to 70.08%. That evidence comes from eye images, not skin. We port the Sinkhorn alignment term to our image classifier, with skin-type band as the group, and give it the same backbone, data and hyperparameter search budget as our method. Beating it means two things at the finest label level: a smaller $\Delta(k)$ with no loss in overall balanced accuracy, and better accuracy on FST V–VI in absolute terms rather than parity bought by lowering the light-skin result.
The key ablations isolate the ingredient we are claiming. The first replaces the perilesional reference with a random distant patch from the same image. The second keeps the second stream but feeds it a global color-normalized image instead of the reference ratio. The third removes the RGB stream. If the proximity ablation performs as well as the full method, the benefit is generic color normalization and not self-referencing, and we would report it that way.
Why not just…?
Why not just use skin-tone color augmentation or color constancy? Randomly shifting hue and brightness, or normalizing every image's color, makes a model indifferent to global color. Our argument is that lesion diagnosis depends on color relative to the surrounding skin, and global augmentation weakens that cue rather than isolating it. We found no reported result on Lacuna for color-constancy or skin-tone augmentation on Fitzpatrick 17k or PASSION, so this is an open empirical question, not a settled one. The normalization ablation and a tuned augmentation baseline settle it directly. If either matches the full method on the $\Delta(k)$ curve, our mechanism adds nothing.
Why not just train on more pigmented-skin data such as PASSION? Diverse data is necessary, and we use it. PASSION's own finding is that a model trained in two countries degrades in a third, so data from one region does not transfer to dark skin in general. Data also cannot recover contrast lost below the noise floor, and our floor prediction distinguishes the two explanations. We run every method with and without pooled PASSION training. If adding data alone flattens $\Delta(k)$ at fine levels, the self-reference contribution shrinks to a cheaper path for data-poor settings, and we say so.
Why not trust the network to learn the reference itself, since it already sees surrounding skin? This is the most serious threat. A convolutional network looking at a whole photo can in principle learn to compare lesion and background. Two experiments test it. The first scales the backbone, where Luo et al. suggest more capacity can widen the gap rather than close it. The second is a full-image baseline at matched compute. If large models given the whole photo already show a flat $\Delta(k)$, the second stream is redundant. The remaining contribution would then be the granularity-scaling analysis and the floor-based abstention.
Why not just abstain on all uncertain cases, as DeepTag does? Abstaining on a fixed fraction of the least confident cases ignores who those cases are. If confidence is miscalibrated on dark skin, a fixed threshold may confidently misclassify exactly the patients the method should protect. We compare our signal-based abstention with DeepTag-style confidence abstention at equal coverage. Success means higher precision on FST V–VI at matched coverage, without abstaining disproportionately on those patients.
Proposed Experiments and Decisive Checks
The first check is the scaling hypothesis itself. We plot $\Delta(k)$ against granularity level for the raw-RGB baselines on Fitzpatrick 17k, with confidence intervals bootstrapped over images. If the curve is flat, the main premise fails, and the proposal reduces to a color-correction trick to be judged on its own merits.
The second check is whether self-referencing flattens the curve and whether proximity matters. It compares the full method against the random-patch and normalization ablations, the alignment baseline and the augmentation baseline, all at matched budgets. The method is evaluated on both Fitzpatrick 17k and PASSION, with PASSION split by country to probe shift between clinics.
The third check tests the physics directly. We stratify test images by the measured lesion-to-reference signal-to-noise. We expect the gain from self-referencing to concentrate where that signal is moderate, and the residual dark-skin gap to concentrate where it is lowest. If the residual gap shows no relation to the signal measure, the Beer–Lambert account is wrong and the floor-based abstention loses its justification.
The final check is triage utility. We report, at matched coverage, the finest label each patient receives and the accuracy of those labels by skin-type band. The comparison is against confidence-based abstention.
Fitzpatrick 17k lacks biopsy confirmation, so all fine-grained conclusions are reported with that caveat. We also re-examine any result that depends on the noisiest conditions using the verified holdout.
References
- Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology With the Fitzpatrick 17k Dataset
- PASSION for Dermatology: Bridging the Diversity Gap with Pigmented Skin Images from Sub-Saharan Africa
- DDI-CoCo: A Dataset for Understanding the Effect of Color Contrast in Machine-Assisted Skin Disease Detection
- Going beyond the means: Exploring the role of bias from digital determinants of health in technologies
- Beyond Accuracy: A Critical Review of Fairness in Machine Learning for Mobile and Wearable Computing
- A Self-Reference Interference Sensor Based on Coherence Multiplexing
- Isolating Specific vs. Non-Specific Binding Responses in Conducting Polymer Biosensors for Bio-Fingerprinting
- DeepTag: inferring diagnoses from veterinary clinical notes
- Performance of the 0-Hour/1-Hour Algorithm for Diagnosing Myocardial Infarction in Patients With Chest Pain in the Emergency Department ― A Systematic Review and Meta-Analysis ―
- FairCLIP: Harnessing Fairness in Vision-Language Learning