Asymmetric Slack Penalties as Low-Threshold Probability Estimators for Rare Outcomes

When a positive outcome is rare, such as ICU mortality at 1 to 5 percent or an active compound in a screening assay, the probabilities that matter for decisions are small. A clinician acts at a 2 percent risk, not at 50 percent. Standard pipelines handle this badly. They train a classifier, then fit a sigmoid or a step function on a held-out set so the scores match observed frequencies (post-hoc recalibration). That held-out set contains few positives, and the fitted curve is shaped mostly by the majority class. This proposal takes a different route. It builds on a classical fact about soft-margin classifiers, which allow some training points to violate the margin at a price per point called a slack variable. If positives and negatives pay different prices, in the ratio $(1-c):c$, the resulting classifier stops approximating "which class is more likely" and instead approximates "is the positive-class probability above $c$?" Training a small family of such classifiers across the decision-relevant band, for example $c$ from 0.1 percent to 5 percent, with a non-linear kernel feature map and a constraint that their boundaries do not cross, yields probability estimates that aim at calibration exactly where decisions are made. It also needs no separate calibration split.

The claim is narrow and testable. For rare outcomes under realistic model misspecification, estimating a few posterior level sets directly with class-asymmetric slack penalties should give better decision-weighted calibration inside the clinical prevalence band than two alternatives: a globally fitted classifier followed by Platt scaling or isotonic regression, and a proper-scoring-rule model with a kernel calibration penalty. The advantage should be largest when positives are scarce, because post-hoc methods must spend those scarce positives on a holdout set.

Core Claim and Mechanism

Let $f_c$ be a function in a reproducing kernel Hilbert space $\mathcal{H}$, the function space induced by a kernel $k$. In practice we approximate $\mathcal{H}$ with random Fourier features so the method scales beyond the $\mathcal{O}(n^2)$ Gram matrix. For a threshold $c \in (0,1)$ we solve

$$\min_{f_c, \xi} \tfrac{1}{2}\|f_c\|_{\mathcal{H}}^2 + C\Big[(1-c)\sum_{i: y_i=+1}\xi_i + c\sum_{i: y_i=-1}\xi_i\Big] \quad \text{s.t. } y_i f_c(x_i) \ge 1-\xi_i,\ \xi_i \ge 0,$$

where $y_i \in \{-1,+1\}$, $\xi_i$ is the slack of point $i$, and $C$ is the usual regularization weight. The population minimizer of this cost-weighted hinge loss is $\operatorname{sign}(\eta(x) - c)$, with $\eta(x) = P(Y=1 \mid X=x)$. This is a known property of weighted hinge losses. We treat it as the method, not as a contribution.

For a grid $c_1 < \dots < c_K$ spanning the decision band, we fit all $K$ classifiers jointly on shared random features. We add non-crossing constraints $f_{c_k}(x_i) \ge f_{c_{k+1}}(x_i)$ at the training points so the implied level sets are nested. The probability estimate at $x$ is read off by locating where the sign flips across the grid and interpolating between adjacent $c_k$. Outside the band the estimate is clamped, and we make no claim about calibration there.

The asymmetric slack does two things at once. It keeps rare positives from being swamped: at $c = 0.02$ each positive's violation costs 49 times a negative's. It also fixes what "correct" means for each classifier. That second role is what turns an imbalance fix into a probability estimator.

Relation to Prior Work

Three groups of papers matter, and they play different roles.

The first group motivates the gap. Learning Confidence Bounds for Classification with Imbalanced Data is the closest work. It adds slack variables to a concentration-bound problem and shifts a pretrained linear threshold toward the majority class, improving F1 and G-Mean. It never produces or evaluates probabilities, and it notes that threshold shifts help little when the representation does not separate the classes. We differ in three ways: we fit a nested family of boundaries rather than a single one, we learn them in a non-linear feature space rather than on a frozen linear score, and we evaluate them as a posterior estimate.

Aligning Evaluation with Clinical Priorities shows that AUC-ROC averages accuracy over label shifts weighted by the model's own score distribution rather than real prevalence. It proposes the Bounded DCA Log Score, which clips cross-entropy to a realistic prevalence range, but only as an evaluation tool. We adopt it as the primary metric. Our training objective can be read as a discretized, margin-based counterpart of the band that score integrates over. The concern that aggregate calibration hides rare-subgroup failure, documented in long-tailed vision by Improving Calibration for Long-Tailed Recognition, motivates restricting attention to a band rather than a global error.

The second group supplies competing methods and constrains the design. Calibration by Distribution Matching adds a kernel distribution-matching penalty to the negative log-likelihood and can target "decision calibration" near chosen thresholds. That is the strongest conceptual competitor, since it also focuses calibration where decisions happen. However, it was evaluated only on balanced tasks, and its penalty's estimate is averaged over mini-batches that, under a 1 percent base rate, contain almost no positives. Trainable Calibration Measures From Kernel Mean Embeddings documents exactly this failure: its calibration penalty degenerates when a batch has no misclassified points. That constrains us to prefer objectives whose signal from each rare positive does not depend on batch composition, which weighted slack provides.

Optimal and Provable Calibration in High-Dimensional Binary Classification proves that in high dimensions classical estimators are overconfident and that Platt scaling reaches the optimal correction only with a large holdout. That paper assumes balanced priors. Our hypothesis is that with rare positives, the effective holdout size is the number of positives in it, which makes Platt's requirement bite much harder. Posterior Re-calibration for Imbalanced Datasets corrects for prior shift post hoc with a tuned exponent $\lambda$. It needs a validation set matched to deployment and cannot change the learned boundary.

The third group consists of margin methods that do not target probabilities. Margin Calibration for Long-Tailed Visual Recognition fits per-class affine margin corrections and reports only accuracy. Calibration-Aware Margin Loss adds absolute margins to equalize operating points in retrieval, but depends on per-class statistics that break on tails. Neither treats margin violations as encoding posterior level sets.

Research Program

We would build a solver for the joint $K$-threshold problem on random Fourier features. It would be a primal stochastic subgradient method with a hinge penalty on crossing violations. We would study three questions.

The first is statistical: how does level-set estimation error at small $c$ scale with the number of positives? The heuristic is that the $c$-classifier's slack weights make each positive carry roughly $1/c$ times the weight of a negative. The effective sample size near the $c$ level set is therefore governed by the positive count, as it is for Platt. The difference is that no positives are diverted into a holdout, and the boundary itself is fit where the posterior crosses $c$ rather than globally.

The second is practical. We would examine how the grid size $K$, the non-crossing constraint, and the kernel bandwidth trade off against one another.

The third is a failure analysis. We would ask when a well-specified kernel logistic regression makes the approach unnecessary. We expect this whenever the model family contains the true posterior, since the proper log loss is then already consistent everywhere.

Baselines

The naive reference point is an unweighted kernel soft-margin classifier on the same random features. Its signed distance is squashed with a logistic function and no recalibration is applied. This shows what the accuracy paradox does to raw scores and sets the floor.

The main post-hoc competitor is class-weighted kernel logistic regression followed by Platt scaling, with isotonic regression as a second variant. Both use a calibration split carved from the training data. This is the standard practitioner pipeline, and Optimal and Provable Calibration in High-Dimensional Binary Classification gives it theoretical backing.

The prior-correction competitor is Imbalance Calibration from Posterior Re-calibration for Imbalanced Datasets, run with its public code, GT-RIPL/UNO-IC. Its $\lambda$ is tuned on the same calibration split.

The strongest existing in-training method on this exact question is the NLL-plus-MMD objective of Calibration by Distribution Matching, configured for decision calibration at the same thresholds $c_1,\dots,c_K$. A second in-training variant uses the MMCE penalty of Trainable Calibration Measures From Kernel Mean Embeddings. For MMD we will also try positive-stratified mini-batches, so that a batch-composition fix is not unfairly withheld from it. For numbers, the file reports no calibration figures on our clinical datasets. On eICU the only reported result is APACHE IV AUC-ROC of 0.907 for African-American patients versus 0.868 for Caucasian patients, from Aligning Evaluation with Clinical Priorities. That is a discrimination number, not a calibration target. On Hepatitis the confidence-bound method reports F1 of 0.535, against 0.485 for the unadjusted baseline and 0.224 for SMOTE (Learning Confidence Bounds for Classification with Imbalanced Data). We found no reported result on Lacuna for Bounded DCA Log Score or band-restricted calibration error on eICU, MIMIC-III, Hepatitis, Pima, or Tox21. All baselines will therefore be rerun by us.

Fairness is defined by total labeled data. Every method sees the same labeled pool, and methods that need a calibration split must take it from that pool. We will also report a "generous" condition in which post-hoc methods get an extra calibration set of increasing size. This measures how many additional positives they need to catch up. All methods share the same random feature map and dimension, and each gets the same hyperparameter search budget, selected by cross-validated Bounded DCA Log Score within the training pool.

Four ablations isolate the key ingredient. The first is a single cost level followed by Platt scaling, which tests whether the grid matters. The second drops the non-crossing constraint, which tests whether nesting matters. The third uses symmetric slack with the full grid read off by Platt, which tests whether asymmetry is doing the work. The fourth uses a grid deliberately placed away from the deployment band, which tests whether targeting matters.

Beating the field means a statistically reliable improvement in Bounded DCA Log Score over the 0.1 to 5 percent band, measured across repeated splits, against both the Platt pipeline and MMD decision calibration. It also means discrimination, measured as AUROC and precision-recall AUC, stays within noise.

Why not just…?

Why not just fit class-weighted kernel logistic regression and apply Platt scaling? This is the most likely threat, and it may win when positives are plentiful or the kernel model is close to well specified. The case against it is specific. Its sigmoid is fit on a holdout whose positives are scarce. The high-dimensional analysis in Optimal and Provable Calibration says that holdout must be large. A single monotone map also cannot fix a boundary that is misplaced near a low threshold. The generous-holdout condition settles the question directly by measuring the extra positives Platt needs to match us. If that number is near zero, the proposal fails.

Why not just train unweighted kernel logistic regression, since log loss is a proper scoring rule? When well specified it is consistent everywhere, and we expect it to tie in that regime. We build a synthetic control in which the true posterior is inside the kernel family, and we expect no gain there. Under misspecification, log loss spends capacity where most probability mass lies, which is the majority region. Band-focused level-set estimation should then win. A second synthetic setting with a deliberately misspecified low-probability region tests this.

Why not just add the decision-calibration MMD penalty of Calibration by Distribution Matching? It targets the same thresholds and needs no holdout, so it is the most direct competitor. Its known weakness is the variance of batch-level kernel estimates when a batch holds zero or one positive, the same failure Trainable Calibration Measures reports for MMCE. We give it stratified batches. If it then matches us, the contribution reduces to an analysis of batch composition, and we will report it that way.

Why not just correct priors post hoc with Imbalance Calibration? A single exponent on prior ratios corrects a global shift but cannot reshape a boundary. It also depends on a validation set that matches deployment (Posterior Re-calibration for Imbalanced Datasets). The prevalence-shift experiment below tests the case it is designed for.

Proposed Experiments

The main benchmarks are binary clinical and scientific tasks from the file. These are eICU mortality with demographic splits, MIMIC-III ICU outcomes, UCI Hepatitis, Pima, Heart Disease, and Breast Cancer, and Tox21 assays, which are naturally rare-positive. We add the synthetic 100:1 Gaussian setting from Learning Confidence Bounds, extended with the well-specified and misspecified controls. To test beyond tabular inputs, we use frozen embeddings from a CIFAR-100-LT backbone and form one-versus-rest tasks for tail classes.

The primary metric is the Bounded DCA Log Score over 0.1 to 5 percent. Secondary metrics are a binless, kernel-smoothed calibration error restricted to predictions in the band, which avoids bin-width artifacts, along with Brier score, AUROC, and precision-recall AUC. On eICU we report each metric per demographic group, because a gain confined to the majority group would not count.

Three checks decide the question. The first subsamples positives while holding negatives fixed. If the proposal is right, its advantage over the Platt pipeline should grow as positives shrink and vanish in the well-specified synthetic control. The second trains at one prevalence and evaluates after resampling the test set to other prevalences within the band. Level-set estimators of $\eta$ should transfer under label shift after the standard analytic prior correction, and here we compare against Imbalance Calibration tuned on a mismatched validation set. The third runs the ablations. If symmetric slack read off by Platt matches the asymmetric grid, the claim that asymmetric penalties carry the calibration signal is refuted.