A common way to build an interpretable clinical risk score is to cut each lab value or vital sign into intervals, then represent each interval as its own on/off indicator. This is one-hot encoding: a patient's lactate falls in exactly one bin, and that bin's indicator is 1 while all others are 0. A linear model then learns one weight per bin, which gives a step-shaped risk curve per measurement. Binarsity made this practical at scale. It adds a penalty on the differences between adjacent bin weights, called a total-variation penalty, which fuses neighbouring bins into flat steps and removes whole irrelevant features. The surviving jumps are the model's learned cut-points. The method was only evaluated on general-purpose UCI datasets, and it has never been examined in the regime that defines clinical prediction: the positive outcome is rare, and the physiologically extreme bins are occupied almost entirely by the few patients who have it.
Our hypothesis is that, in this regime, a penalty that fuses adjacent bins works against the clinically important signal. The jump from "high" to "very high" lactate is exactly where risk changes. Yet that jump is supported by few training rows, and most of those rows are minority-class rows. Under a loss dominated by the majority class, the likelihood gained by keeping the jump is small next to the penalty paid for it. The optimizer therefore flattens the tail and erases the rare threshold. The result would be a model with acceptable AUROC but poor precision on the rare positives, which matches the pattern MIMIC-Extract reports for ICU tasks: AUROC of 0.85–0.94 with AUPRC "often below 0.50". We propose to measure whether this erasure happens and to test a small change to the penalty that prevents it. We will also check, honestly, whether simply reweighting the classes already fixes it.
Core Claim
Under severe outcome imbalance, the fusion penalty in binarsity systematically removes cut-points in the sparsely populated, minority-dominated tails of clinical variables. We claim that making the penalty weight on each adjacent-bin jump depend on how much class-balanced evidence supports that jump recovers those cut-points. We expect this to improve AUPRC and tail calibration over both plain and class-weighted binarsity, while keeping the model a sparse, readable step-function score. Concretely, we fit
$$\min_{\beta}\ \frac{1}{n}\sum_{i=1}^{n} \omega_{y_i}\,\ell(y_i, x_i^\top\beta) \;+\; \lambda\sum_{j}\sum_{k} w_{jk}\,\lvert\beta_{j,k+1}-\beta_{j,k}\rvert,$$
subject to binarsity's per-feature linear constraint on the bin weights. In this objective, $x_i$ is patient $i$'s one-hot bin vector, $\beta_{j,k}$ is the weight of bin $k$ of feature $j$, and $\ell$ is the logistic loss. The term $\omega_{y}$ is a class weight: inverse class frequency, or 1 for the unweighted variant. The parameter $\lambda$ sets the overall penalty strength. The per-jump weight $w_{jk}$ is our change. It scales with the square root of the class-weighted number of training rows lying on the tail side of cut $k$, rather than with the unweighted count. The square root comes from the usual scaling of a coefficient's standard error with the amount of data supporting it. A tail supported by 40 patients, 30 of them positive, is then penalized according to its balanced evidence rather than its raw headcount. A second, label-informed change places bin edges using quantiles of the pooled class-conditional distributions instead of fixed equal-count bins. This responds to the fixed $d_j = 50$ bin count that binarsity used throughout. All label-dependent choices are computed inside training folds only.
Research Program
The work proceeds in three steps.
The first step is diagnostic. On simulated data with planted thresholds, we generate clinical-like long-tailed covariates and a rare outcome whose log-odds jump at known tail cut-points. We vary the imbalance ratio from 1:1 to 1:200. We then record, for plain and class-weighted binarsity, which planted cut-points survive as fitted jumps. If the erasure hypothesis is wrong, tail cut-points will be recovered about as often as central ones at every imbalance ratio, and the project reduces to a negative result worth reporting.
The second step changes the penalty as above. It keeps the existing solver structure, since binarsity's reported runtime of about 2–5 times standard Lasso is part of its appeal. Only the weights change, so the proximal step is unchanged.
The third step tests the change on real ICU data and on controlled re-imbalancing of standard tabular datasets.
Prior work plays three different roles here. Binarsity supplies the method we modify, and its untested behaviour under skew is the gap we address. MIMIC-Extract motivates the problem, because it documents that high AUROC coexists with low AUPRC on ICU events. It also supplies the benchmark and the tree-ensemble reference point. MyESL pairs one-hot sequence features with sparse group penalties and shows that ignoring imbalance lowers minority prediction confidence. That supports our premise that sparse one-hot penalization interacts with class skew. Its fix, though, is choosing negatives with a phylogenetic tree, and that has no counterpart for a population-level ICU risk score, where the deployment population cannot be curated. Imbalance Trouble shows that in deep networks the classifier weights for minority classes shrink in norm under imbalance. Our question is a linear, structured analogue: the minority-supported coefficient differences are the ones pushed to zero. We borrow its intuition, not its theory, since its analysis assumes an unconstrained deep feature model rather than a penalized generalized linear model. Encoding High-Cardinality String Categorical Variables limits the scope. One-hot encoding breaks down for messy, high-cardinality strings, so we restrict the study to binned continuous measurements and clean categorical codes, where one-hot encoding remains the standard.
Baselines
The naive reference point is unweighted logistic regression on the raw, standardized continuous features. It can only represent monotone linear risk and cannot express a tail threshold. Any binned model should beat it on AUPRC, and if it does not, binning is not worth its cost on these tasks.
The standard one-hot baselines are Lasso on the same bins and unmodified binarsity, run with the original fixed bin count. These isolate what the structured penalty already buys.
The key ablation is class-weighted binarsity: inverse-frequency $\omega_y$ with the original, unweighted $w_{jk}$ and the original bins. This is the most important comparison in the study, because it is what a careful practitioner would do first. Our method must beat it on tail cut-point recovery in simulation and on AUPRC in the real data. A tie means the evidence-scaled jump weights add nothing. A further ablation separates the two changes, the reweighted jumps and the label-informed bin edges, so that each can be credited independently.
The strongest existing competitor is a Random Forest, with gradient boosting alongside it. On MIMIC-Extract intervention onset and offset prediction, the reported macro AUROC is 0.90–0.94 for Random Forests and LSTMs, and mortality AUROC is 0.85–0.90 across models (MIMIC-Extract). AUPRC is reported only as "often below 0.50", with no per-task value. We found no reported AUPRC per task and no binarsity result on MIMIC-Extract on Lacuna, so all baselines will be rerun by us.
Fairness conditions are the same for every method. All models use the same MIMIC-Extract hourly aggregates summarized over the same observation window and the same patient-level splits. Each method gets an equal hyperparameter search budget: the same number of configurations, selected by validation AUPRC. Tree ensembles also get class weighting. We count the proposal as beating the forest if it matches its AUPRC within bootstrap confidence intervals while using far fewer active features and exposing explicit thresholds. We count it as beating class-weighted binarsity only if the AUPRC gain is significant across bootstrap resamples and survives repeated splits.
Why not just…?
Why not just use class-weighted loss with standard binarsity? This is the real threat, and it may win. Reweighting raises the likelihood contribution of positive rows, which could already make tail jumps worth keeping. It does not change the fact that a tail bin holds few rows in absolute terms, so the penalty can still dominate at the $\lambda$ that validation selects for the whole model. The simulation settles this directly. If class-weighted binarsity recovers planted tail cut-points as well as our variant at every imbalance ratio, we drop the jump reweighting and report that reweighting suffices.
Why not just use gradient boosting or random forests? On MIMIC-Extract, Random Forests already match LSTMs on AUROC (MIMIC-Extract), and they may also win on AUPRC. We do not claim to beat them outright. Binarsity is reported to match forests and boosting on UCI data while running far faster (Binarsity). Our claim is that a sparse step score with explicit, bedside-checkable thresholds can reach forest-level AUPRC. If it falls clearly short, that is an important finding: interpretability would be costing rare-event precision.
Why not just oversample positives or use SMOTE? Duplicating minority rows is equivalent in expectation to class weighting for a linear model, so it is covered by the key ablation. SMOTE's interpolated rows land in intermediate bins, which smears tail evidence across neighbouring bins rather than concentrating it at the true cut. We include SMOTE-plus-binarsity as an extra arm and measure whether it moves the recovered cut-points away from the planted ones in simulation.
Why not just skip binning and use splines or raw values? Additive spline models can represent tail risk too. However, their smoothness penalty is also dominated by where the data are dense, so the same erasure question arises. We include a class-weighted additive spline logistic model as an arm and compare its tail calibration to the step models.
Proposed Experiments
The first experiment is the simulation described above. It uses planted tail and central thresholds across imbalance ratios from 1:1 to 1:200. We report cut-point recall and localization error for tail versus central thresholds, with 50 replicates per setting. The decisive check is whether tail recall under class-weighted binarsity falls with imbalance while central recall does not, and whether our weighting closes that gap.
The second experiment uses MIMIC-Extract in-hospital mortality, prolonged length of stay, and vasopressor and ventilation onset. Every baseline above is run under the shared protocol. We report AUPRC, sensitivity at 90% specificity, and calibration error computed separately within the top and bottom 5% of each key lab's range. We also report the number of active features and the bootstrap stability of recovered cut-points.
The third experiment re-imbalances Adult and Bank marketing, where binarsity's behaviour is established, by subsampling positives to fixed ratios. This tells us whether any gains are specific to clinical tails or general to skewed tabular data.
The study would be falsified in either of two ways. The first is that tail thresholds are not erased even at 1:200 imbalance. The second is that class-weighted binarsity recovers them as well as our variant does. Either outcome would redirect the work toward explaining why the low MIMIC-Extract AUPRC comes from somewhere other than the discretization.
References
- Binarsity: a penalization for one-hot encoded features
- MIMIC-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC-III
- MyESL: A Software for Evolutionary Sparse Learning in Molecular Phylogenetics and Genomics
- Imbalance Trouble: Revisiting Neural-Collapse Geometry
- Encoding high-cardinality string categorical variables