Focal loss trains a classifier to pay more attention to examples it currently gets wrong. That is exactly what helps rare classes, such as a tiny adrenal gland in an abdominal scan or a rare land-use category in satellite imagery. The same rule also hands the largest weight to examples that are wrong for bad reasons: a mislabeled voxel, a hospital's idiosyncratic annotation convention, or a scanner artifact present at only one training site. When training data pool several hospitals or regions, the model then spends its effort memorizing site-specific quirks that will not transfer to a new site. We propose a simple test that uses the site label every multi-site dataset already carries. An example earns focal upweighting only if it is about as hard as examples of the same class at the other training sites. If it is much harder than anything other sites show for that class, its weight is smoothly flattened instead of amplified.
The bet is that "hard everywhere" is the signature of genuine minority-class difficulty, while "hard only here" is the signature of noise or site-specific shift. Separating the two needs no clean held-out set and no extra network. It only needs running statistics of per-class losses broken down by site. We will test the idea mainly on multi-scanner abdominal segmentation, where small organs collapse under scanner shift, and secondarily on cross-region satellite classification, where the same tension appears with geography as the grouping variable.
Core Claim
Two literatures pull a sample-weighting function in opposite directions. Focal Loss for Dense Object Detection multiplies cross-entropy by $(1-p_t)^\gamma$, where $p_t$ is the predicted probability of the true class, so that hard examples dominate. Noisy-label work goes the other way. Learning Adaptive Loss for Robust Learning with Noisy Labels observes that well-tuned robust losses flatten automatically on large-error samples as noise rises. Improve Noise Tolerance of Robust Loss via Noise-Awareness learns per-sample loss hyperparameters with a small MLP. Both of those methods need a verified clean meta-set and look only at each sample's own margin. Threshold-adaptive Unsupervised Focal Loss for Domain Adaptation of Semantic Segmentation deliberately keeps gradients alive on low-confidence pixels, and is therefore exposed to exactly the noise these papers warn about.
Our claim is that, on multi-site data, the site label resolves this conflict without a clean set. Hardness that recurs across sites should be focused on. Hardness that exceeds what other sites show for that class should be flattened. We further claim that this specific grouping, the leave-own-site-out comparison, is what does the work. A per-class version that ignores sites, or a per-site version that ignores classes, should not reproduce the gain.
The Loss
For a training example $i$ (an image, or a voxel in segmentation), let $y_i$ be its label, $d_i$ its site (hospital, scanner, or geographic region), and $\ell_i=-\log p_t$ its cross-entropy. During training we keep exponential-moving-average loss histograms for every (class, site) cell. From these we read off two quantities:
- $\tau_{y,-d}$, the $q$-quantile of losses for class $y$ pooled over all sites except $d$;
- $s_{y,-d}$, the interquartile range of that same pooled distribution.
The training loss is
$$ L_i = \mathrm{sg}\!\left[(1-p_t)^{\gamma}\,\exp\!\left(-\frac{\max(0,\ \ell_i-\tau_{y_i,-d_i})}{s_{y_i,-d_i}}\right)\right]\ell_i , $$
where $\mathrm{sg}$ stops gradients through the weight, $\gamma$ is the usual focal exponent, and $q$ sets how permissive the gate is.
The first factor in the weight is ordinary focal upweighting. The second factor is 1 for examples no harder than the bulk of their class elsewhere, and decays for examples beyond that. Because the decay scale is read from the data rather than tuned, this inherits the "learn the scale, don't fix it" spirit of adaptive robust losses. In segmentation, the loss is added to the usual Dice term unchanged.
Some cases need explicit handling. Classes absent from all other sites fall back to plain focal loss. This fallback is a stated limitation, because a genuinely site-unique subpopulation (for example, the only pediatric site) will be treated as unverifiable. With few sites, the quantiles are estimated from pooled voxel losses rather than per-image means, so even two or three other sites give stable histograms in dense tasks.
Research Program
The work has three parts.
The first part is a diagnosis that should precede any method claim: does focal loss actually concentrate weight on site-specific error? On each benchmark, we train with plain focal loss. We then measure what fraction of total focal weight lands on examples whose loss is above the other-site quantile, and how that fraction evolves over epochs. We repeat this after injecting synthetic label noise into one training site only. If high-weight examples are not disproportionately site-idiosyncratic, the premise fails and we stop.
The second part implements the gated loss as a drop-in replacement in a standard 3D U-Net pipeline and a standard image classifier. The overhead is a histogram update per batch.
The third part is a set of ablations that isolate the grouping. We compare:
- class-only gating, which uses the loss quantile of class $y$ over all sites, own site included;
- site-only gating, which uses the quantile over all classes at other sites;
- shuffled-site gating, which uses the same machinery with site labels randomly permuted.
If class-only gating matches the full method, the site label adds nothing, and the honest conclusion is that a per-class robust cap on focal loss is enough.
Baselines
The naive reference is ERM with standard cross-entropy, plus Dice for segmentation, with the default augmentation of each pipeline. WILDS: A Benchmark of in-the-Wild Distribution Shifts shows this is hard to beat. Its specialized robust objectives (CORAL, IRM, Group DRO) failed to meaningfully outperform it on 8 of 10 datasets. On Camelyon17-WILDS it reports roughly 93% in-distribution and about 70% out-of-distribution accuracy for ERM. On AMOS, AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation reports 89.04% mDice for 3D-UNet on the CT test set (above Swin-UNetr's 86.32%) and a drop of over 20% mDice on unseen MRI scanners. The 3D-UNet number is the reference we must not fall below in-distribution. For FMoW-WILDS, no reported result was found on Lacuna, so we will run ERM ourselves under the official splits.
The strongest imbalance competitor is static focal loss with $\gamma$ grid-searched, combined with inverse-frequency class weights. Each model gets the same grid size as our method's $q$ and $\gamma$ search, selected on the benchmark's official in-distribution validation split. Selecting on in-distribution data is the only fair option, since no target labels exist.
The strongest calibration-oriented adaptive focal method is AdaFocal: Calibration-aware Adaptive Focal Loss. It adjusts $\gamma$ per confidence bin from validation feedback and reports CIFAR-100 ResNet-50 ECE of 1.36% against 5.64% for focal loss. We run it with the same validation split it would have in practice.
The strongest shift competitor is Group DRO from WILDS, which upweights whole high-loss sites. WILDS reports that it raised CivilComments worst-group accuracy from 56.0% to 70.0%. It is the natural alternative use of the same site labels.
For noise, we include generalized cross-entropy as a metadata-free robust loss. We also include the NARL adjuster, given the small clean meta-set it requires, as an upper-bound-style comparison that uses information our method does not.
All methods share backbone, schedule, augmentation and number of tuning trials. Beating the field means two things on the held-out site. First, a gain on small-structure Dice (adrenal glands, duodenum) or on minority-class macro-F1 that exceeds seed-to-seed spread across at least three seeds. Second, in-distribution performance within one point of tuned static focal loss.
Why not just…?
Why not just grid-search a fixed $\gamma$ with class weights? A fixed $\gamma$ cannot tell a rare-but-clean voxel from a mislabeled one; both have low $p_t$ and both get the same boost. AdaFocal's results show that one global $\gamma$ leaves localized miscalibration, though on non-shifted CIFAR-100. The single-site noise-injection experiment tests the claim directly. Tuned static focal loss should degrade as noise at one site rises, while gated focal loss should not. If it does not degrade, this baseline wins, and we will say so.
Why not just use Group DRO, since it also uses site labels? Group DRO acts in nearly the opposite direction. It raises the weight of the hardest site as a whole, which includes that site's noise and artifacts, and WILDS found it gave little OOD benefit on most datasets. Our gate leaves site weights alone and acts only within a class, on examples that are anomalous relative to other sites. Running both with identical site labels shows which use of the metadata transfers to an unseen scanner or region.
Why not just cap the loss per class, ignoring sites? This is the real threat, because a per-class robust cap on focal loss might capture most of the benefit with less machinery. The class-only and shuffled-site ablations exist to settle it. The method's distinctive claim survives only if leave-own-site-out gating beats both on held-out-site small-structure Dice, with the largest gap under single-site noise.
Why not just use a stronger backbone or foundation model? AMOS shows transformers did not beat a 3D-UNet on CT and that both architectures lose over 20% mDice on unseen MRI scanners. Beyond the Prompt: Deploying Medical Foundation Models on Diverse Chest X-ray Populations finds foundation-model zero-shot performance swings by up to 46% MCC across prompt templates, with an adult-to-pediatric shift persisting. The loss is backbone-agnostic, so we also apply it on top of Swin-UNetr to check that gains are additive.
Proposed Experiments
The main testbed is AMOS. We hold out scanners in two settings: one CT scanner at a time, and the paper's unseen-MRI protocol using joint CT+MRI training. Domain $d$ is the scanner. We report per-organ Dice with emphasis on adrenal glands and duodenum, mDice, and the in-distribution/held-out gap.
Camelyon17-WILDS uses hospital as $d$. It serves as a control: its binary tumor label is close to balanced, so any gain there must come from flattening site-specific noise rather than from imbalance. We report OOD test accuracy against ERM's ~70%.
FMoW-WILDS uses world region as $d$, to test whether the mechanism transfers to geographic shift with a long-tailed label set. We report worst-region and macro accuracy.
Each benchmark also gets a variant with synthetic label noise injected into a single training site. On AMOS, the noise is boundary erosion and dilation of small organs, mimicking annotation-convention differences. Across all benchmarks, we report expected calibration error on the held-out site, because AdaFocal's results suggest focal variants trade accuracy against calibration.
Three outcomes would decide the question:
- Gated focal loss beats tuned static focal loss and Group DRO on held-out-site small-structure Dice in AMOS, with the margin widening under single-site noise.
- Class-only and shuffled-site gating recover substantially less of that margin.
- The diagnostic shows plain focal loss concentrating weight on above-quantile, site-idiosyncratic examples.
If the first result holds but not the second, the contribution shrinks to "robustly capped focal loss," which is a smaller and still useful finding. If none holds, the premise that focal loss amplifies site-specific error under shift is wrong for these benchmarks.
References
- Focal Loss for Dense Object Detection
- Learning Adaptive Loss for Robust Learning with Noisy Labels
- Improve Noise Tolerance of Robust Loss via Noise-Awareness
- Threshold-adaptive Unsupervised Focal Loss for Domain Adaptation of Semantic Segmentation
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- AdaFocal: Calibration-aware Adaptive Focal Loss
- Beyond the Prompt: Deploying Medical Foundation Models on Diverse Chest X-ray Populations