Many practitioners train classifiers with losses such as focal loss, which down-weights easy examples to handle class imbalance. These losses are known to bend the model's confidence scores away from true probabilities. Recent theory says the bending is predictable. For a large family of such losses, a fixed formula (the Ψ-transform) maps the model's raw outputs back to honest probabilities, with no extra data required. That claim has so far been checked only on a synthetic three-class toy problem. This proposal tests it on real vision, text, and medical benchmarks. It also asks a sharper question the theory cannot answer by itself: how much of a deep network's miscalibration comes from the choice of loss, which the formula can remove for free, and how much comes from overfitting on finite data, which only a calibrator fitted on held-out data can fix?
The practical stake is concrete. If the loss-derived correction removes a large, well-defined share of the miscalibration, it becomes a zero-parameter calibrator. It would be useful exactly where fitted calibrators struggle: when few labeled validation examples exist, and when the class balance at test time differs from training. It would also give fitted calibrators a better starting point. If it removes almost nothing beyond what temperature scaling already fixes, then the synthetic evidence was misleading, and the literature should say so.
Core Claim
Calm Composite Losses: Being Improper Yet Proper Composite introduces a "calmness" condition. A loss is calm if its slope diverges to minus infinity as the predicted probability of the true class goes to zero. The paper proves that any calm loss is secretly strictly proper composite, meaning that a specific inverse link, the Ψ-transform, turns its minimizer into the true class probability. Focal loss and generalized cross-entropy both satisfy calmness. The paper's only evidence is synthetic data, where Ψ drives the KL divergence to the true posterior to "nearly zero" without changing classification error.
Our claim is that, on real networks, the Ψ-transform removes the population-level distortion of the loss but not the finite-sample overconfidence that all deep models share. Because the two effects have different shapes, a single temperature cannot capture both. The focal-loss distortion is a nonlinear map on probabilities, while overconfidence is roughly a rescaling of logits. We therefore predict three things. First, applying Ψ followed by temperature scaling beats temperature scaling alone on focal-trained models, with the gap largest when calibration data is scarce. Second, Ψ alone beats a fitted calibrator when the test class prior shifts. Third, both effects are visible under decision-theoretic metrics, not just under binned calibration error.
A negative result on any of these three is informative and publishable. It would mean real calibration failure is dominated by overfitting, so the calmness theory has little practical reach.
Research Program
Part 1: implementing the links. We implement Ψ-transforms for binary and multiclass focal loss (over a grid of focusing parameters γ) and for generalized cross-entropy. We verify each implementation by reproducing the synthetic result of Bao and Charoenphakdee on multivariate Gaussian data. On that data we know the true posterior, so we can measure KL divergence to the truth directly.
Part 2: a controlled bridge from synthetic to real data. Still on synthetic Gaussians, we grow model capacity from a well-specified linear model to overparameterized networks, and shrink the training set. For each setting we measure two quantities against the known posterior: the gap closed by Ψ, and the gap left over. This probes exactly the misspecification and finite-sample regime that Consistency and Finite Sample Behavior of Binary Class Probability Estimation assumes away when it bounds probability error by the square root of excess risk.
Part 3: real benchmarks. We move to real data, where the true posterior is unknown. There we score calibration the way Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration recommends. That paper shows binned calibration error can be badly wrong: on SITW speaker verification, it reports an expected calibration error of 0.2 alongside a relative calibration loss of 16.7%. The metrics it recommends are:
- Expected proper scoring rules: log loss and Brier score, which reward both calibration and discrimination.
- Calibration loss (CalLoss): the drop in expected proper score after a reference calibrator is fitted.
- Relative calibration loss: the share of total loss attributable to miscalibration.
These metrics let us define the decomposition cleanly on real data. The share of CalLoss eliminated by Ψ alone is the "loss-induced" part. The share eliminated only by a subsequent fitted calibrator is the "finite-sample" part.
Handling the known weakness of CalLoss. The same paper warns that CalLoss underestimates miscalibration if the reference calibrator is too weak. To address this, we fit the reference calibrator on a large held-out split with a flexible family, and report sensitivity to that choice.
Baselines
Naive reference points. The first is the raw focal-loss model, with its sigmoid or softmax outputs read as probabilities; this is what practitioners do today. The second is the same architecture trained with plain cross-entropy. Cross-entropy is strictly proper, so it has no loss-induced distortion. It shows how much of the calibration problem exists even when the loss is already correct.
Strongest post-hoc competitor. This is temperature scaling applied to the focal model's logits, alongside the direction-preserving calibration that Ferrer and Ramos evaluate. They found temperature scaling fails under prior shift and recommended direction-preserving transforms in its place.
Fairness rules for all post-hoc methods:
- Every calibrator sees the same held-out split.
- We sweep the calibration-set size from a few dozen examples to thousands.
- All methods get the same tuning budget.
- Ψ uses no calibration data at all.
Ψ "beats" temperature scaling if Ψ followed by temperature scaling yields lower log loss and CalLoss than temperature scaling alone, with the improvement exceeding seed-to-seed variation across five training seeds. Ψ alone "beats" fitted calibration under prior shift if it reaches lower log loss than temperature scaling when the test class proportions differ from the validation proportions.
Strongest in-training competitor (multi-label). This is the Strictly Proper Asymmetric (SPA) loss with the Label Pair Regularizer from Towards Calibrated Multi-Label Deep Neural Networks. That work derives a new proper loss for imbalanced labels rather than rescuing an existing one. On WIDER Attribute it reports roughly a fourfold reduction in Average Calibration Error relative to binary cross-entropy while matching mean average precision; the file gives no absolute value. We rerun SPA with its published settings on the same backbone and data splits, then score it under both its own metric and ours.
A focal model plus Ψ counts as competitive if it lands within seed variance of SPA on CalLoss and mean average precision. This matters because it would mean the extra machinery of a newly designed loss is unnecessary. A win for SPA under CalLoss is also informative: it would show that designing properness into training beats restoring it after the fact.
Ablation isolating the key ingredient. We apply a deliberately misspecified Ψ, derived for the wrong γ, and an identity link. If the benefit of Ψ comes from matching the loss's actual distortion, the gains should shrink roughly in proportion to how badly γ is mismatched. If the gains survive a wrong link, then Ψ is just acting as a generic smoothing function.
Why not just…?
Why not just use temperature scaling on the focal model? This is the real threat, and the experiments are built to settle it. Temperature scaling can only rescale the logits, while the focal distortion bends probabilities nonlinearly. That argument alone does not prove the residual matters in practice. The calibration-set-size sweep will show whether temperature scaling alone reaches the log loss of Ψ plus temperature scaling. If it does at every size, we will report that Ψ has no practical value on these benchmarks. Separately, Ferrer and Ramos already found temperature scaling fails under prior shift, and our prior-shift experiment tests whether the data-free Ψ avoids that failure.
Why not just train with cross-entropy and skip focal loss entirely? Practitioners choose focal loss for its effect on accuracy and mean average precision under imbalance, so this is a trade-off, not a free switch. Cross-entropy is one of our baselines on every benchmark. If it matches focal loss plus Ψ on both discrimination and log loss, the honest conclusion is that the rescue is unnecessary on that task. That result would itself be useful, as evidence that the imbalance motivation for focal loss is weaker than assumed.
Why not just design a proper loss from the start, as SPA does? SPA requires deriving and adopting a new objective. Ψ instead leaves existing training pipelines and trained checkpoints untouched. The head-to-head comparison on WIDER Attribute and MS-COCO measures what the rescue route gives up. Cheng and Vasconcelos argue that finite-sample effects need their pairwise regularizer beyond properness alone. If they are right, SPA should win on CalLoss even against Ψ followed by temperature scaling.
Proposed Experiments
Synthetic data. Multivariate Gaussian problems (2 and 10 classes), where the true posterior is known. We measure KL divergence to the truth under raw focal outputs, Ψ, temperature scaling, and Ψ plus temperature scaling, across the capacity and sample-size grid from Part 2.
Single-label real data. CIFAR-10 with ResNet-20 and RepVgg-a2, which Ferrer and Ramos already found are ranked differently by ECE and by proper scores. Also SST2 and AGNEWS with a fine-tuned text encoder, and MedMNIST PATH and PNEUM. On these we report:
- log loss, Brier score, CalLoss, and relative calibration loss;
- accuracy;
- ECE, included only to show where it disagrees with the proper scores.
Prior shift. On MedMNIST and CIFAR-10 we reweight the test class proportions away from the training proportions and compare Ψ alone with each fitted calibrator.
Multi-label real data. WIDER Attribute and MS-COCO, with per-label binary focal loss plus Ψ against binary cross-entropy and SPA with the Label Pair Regularizer. We report mean average precision, Average Calibration Error, and per-label CalLoss. We also reuse the conjunctive retrieval queries of Cheng and Vasconcelos, because errors in the probability of each label multiply as queries combine more labels.
Decisive checks. Four results would settle the question. First, whether the loss-induced share of CalLoss, measured by Ψ alone, is materially above zero on real benchmarks. Second, whether Ψ plus temperature scaling beats temperature scaling at small calibration-set sizes. Third, whether Ψ holds up under prior shift where temperature scaling degrades. Fourth, whether the benefit collapses when the misspecified-γ link replaces the correct one. If the first check fails, the calmness theory does not explain real-world calibration failure, and we will state that plainly.