Threshold-Wise Stacking for Ordinal Prediction

Many prediction targets are ordered grades: cancer stage, essay score, wine quality, lesion risk. When several different models are available for such a task, practitioners usually combine them in one of two ways. They average the models, or they use stacking, which trains a small second-stage model (the "meta-learner") on the base models' held-out predictions so it can learn whom to trust. Every stacking meta-learner we found treats the grades as unordered classes. That choice wastes parameters and can distort the combined probabilities. It also cannot express the kind of trust that ordinal problems seem to call for, such as "rely on the radiomics model to separate no-risk from low-risk, and on the deep network to separate low-risk from high-risk."

We propose to stack ordinal models one boundary at a time. Each base model's predicted distribution is rewritten as a chain of binary questions of the form "given the grade is at least $k$, is it above $k$?" For each boundary $k$, a tiny logistic combiner learns how much to trust each base model on that boundary. The per-boundary weights are shrunk toward a shared set. Multiplying the chain back together always yields a valid, ordered distribution. The claim to test is that this combiner beats averaging, greedy selection, and nominal stacking on ordinal metrics and calibration. If the boundary-specific trust turns out to be empty, the shrinkage should reduce the method to a plain ordinal combiner, and our experiments are designed to detect exactly that case.

Core Claim

The stacking literature fixes the meta-learner's inputs and loss without regard to order. Issues in Stacked Generalization showed that the meta-learner should see class probabilities rather than hard labels, and that multi-response linear regression is a strong combiner, beating majority vote in 8 of 10 comparisons. That study treated even Abalone (age in rings) as an unordered problem scored by error rate. Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap found that logistic-regression stacking sharpens decisions but has the worst log-loss rank among ensembles. The closest ordinal work, Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification, stacks 16 base models for a three-grade pancreatic risk target. Its meta-learner is a nominal XGBoost or random forest, and the authors needed synthetic Gaussian meta-features to keep it from overfitting 678 patients.

Our claim is that the right combiner for ordinal stacking has three properties. It operates in conditional-boundary space. It has one weight per base model per boundary. It shrinks those weights toward each other. Together these properties give three things at once: guaranteed ordered outputs, roughly $B(K-1)$ parameters instead of the $BK \cdot K$ of nominal logistic stacking (for $B$ base models and $K$ grades), and the ability to trust different models at different parts of the scale.

Mechanism

For base model $b$ and grade boundary $k$ ($1 \le k < K$), define $q_{b,k}(x)$ as the model's probability that the grade exceeds $k$ given that it is at least $k$. This is computed from its out-of-fold predicted distribution as $P_b(Y>k)/P_b(Y>k-1)$, where $Y$ is the true grade and $x$ the input. The combiner for boundary $k$ is

$$\text{logit}\, q_k(x) = a_k + \sum_{b=1}^{B} w_{b,k}\, \text{logit}\, q_{b,k}(x),$$

where $a_k$ is a boundary intercept and $w_{b,k}$ is the trust placed in model $b$ at boundary $k$. Each boundary is fit by logistic loss on the out-of-fold samples with $Y \ge k$, which is the conditional decomposition behind the CORN loss. The fit adds a penalty $\lambda \sum_{b,k} (w_{b,k} - \bar w_b)^2$, where $\bar w_b$ is model $b$'s shared weight and $\lambda$ is chosen by nested cross-validation.

The final distribution is recovered as $P(Y>k \mid x) = \prod_{j \le k} q_j(x)$. This product is monotone in $k$ by construction, so no crossing-threshold repair is needed. As $\lambda \to \infty$ the model collapses to a single trust weight per base model. That limit is our key ablation.

Because each boundary's combiner is a low-dimensional logistic model with well-conditioned inputs, we expect it to sharpen probabilities less than nominal logistic stacking does. This expectation is a hypothesis to test, not a guarantee.

Base models are deliberately drawn from different ordinal families so that their errors differ. The tabular-foundation-model study measured a pairwise Q-statistic of 0.961 and only +0.18% accuracy from cascade stacking at 253× compute, which shows how little a homogeneous pool offers. Our pool therefore spans five families:

  1. Integer regression with learned cutoffs, as used for AQI and for staging in Pretrained Vision Models for Predicting High-Risk Breast Cancer Stage.
  2. A cumulative link model with fixed, evenly spaced thresholds. Neural Collapse in Cumulative Link Models for Ordinal Regression reports that fixed thresholds regularize better than learnable ones on UTKFace.
  3. A CORN-trained network, which A New Benchmark for Automatic Essay Scoring in Portuguese found better than cross-entropy and MSE.
  4. A distribution-distance tree model in the style of DK-NODE from Ordinal Regression Based on the Distributional Distance for Tabular Data.
  5. A nominal gradient-boosted classifier.

These families supply the method. The stacking papers motivate the problem and also constrain the design: they rule out high-capacity meta-learners at small sample sizes.

Research Program

We will build the combiner and a shared stacking harness. The harness uses identical outer splits, 5-fold out-of-fold base predictions, and nested selection of $\lambda$. We then run three studies.

Synthetic study. We construct ordinal data where model A is accurate only at low boundaries and model B only at high ones. This checks that the combiner recovers boundary-specific trust and that the shared-weight limit cannot.

Main comparison. We run the full comparison on the tabular and text benchmarks.

Small-sample study. We subsample training sets to a few hundred points, matching the regime of the IPMN study. This tests whether shrinkage alone controls meta-learner overfitting without the synthetic meta-feature augmentation that paper required.

We also include fixed-cutoff versus learned-intercept versions of $a_k$. This tests, at the meta level, whether the fixed-threshold finding from the neural-collapse paper carries over.

Baselines

Best single base model and uniform average. The naive reference is the best single base model chosen on validation, together with uniform averaging of base distributions. Selective averaging is a real competitor: in the breast-staging paper, averaging only models with MSE below 1.0 cut MSE from 0.8297 (best single EfficientNet) to 0.5543. Both are run on the same base pool and folds.

Greedy forward selection. This is ensemble selection with replacement, optimized on validation ranked probability score. The tabular-foundation-model study found it matched far costlier stacking while avoiding the calibration trap, so it is the strongest cheap competitor.

Nominal stacking. We stack with multi-response linear regression and with multinomial logistic regression over all $B \times K$ probabilities, following Ting and Witten. For fairness, its output is decoded with the same expected-cost rule we use, and it receives the same nested regularization budget.

Proportional-odds stacker. This is our method with $\lambda \to \infty$, a single shared weight per model. It isolates the boundary-specific ingredient.

Strongest per-dataset reference. On Eucalyptus, DK-NODE reports accuracy ≈ 0.709 versus ≈ 0.672 for XGBoost. On AES-ENEM, BERTimbau with CORN reports QWK from 0.26 to 0.59 across competencies. On Cyst-X, the flagship random-forest stack reports QWK = 0.595 and binary AUC = 0.839. We found no reported result on Lacuna for any stacking ensemble on WineQuality, Abalone, or Eucalyptus under ordinal metrics.

What counts as winning. Beating a baseline means a lower mean absolute error and a lower ranked probability score than the strongest of greedy selection and nominal stacking. The improvement must hold across at least 10 outer splits per dataset and be significant under a paired test, with expected calibration error no worse than greedy selection.

Why not just…?

Why not just average, or greedily select, the base models? Averaging assigns one weight per model across the whole scale, so it cannot route trust by grade. Greedy selection inherits the same limitation. On datasets where base models are uniformly ranked, we expect greedy selection to tie with us, and we will report that honestly. The synthetic study and the per-boundary weight dispersion on real data show whether the routing capacity is ever used.

Why not just use standard logistic-regression stacking and decode ordinally afterward? This option has more parameters and ignores order during training. The tabular-foundation-model study shows it is the worst-calibrated ensemble on 153 tasks. Post-hoc decoding fixes the point prediction but not the probabilities, so we report ranked probability score and expected calibration error rather than only QWK. If temperature-scaled logistic stacking matches us on all metrics, our representation adds nothing beyond regularization, and we will say so.

Why not just regress on each model's expected grade and round? This is a linear stack of expected values, the scalar summary TRACT optimizes. It assumes equal spacing between grades. The essay-scoring benchmark found MSE regression inferior to CORN, and DK-NODE beat plain trees. We include it as a baseline. Uneven category widths, as in WineQuality where the extreme grades are rare, are where it should fail.

Why not just the shared-weight ordinal stacker? This is the most serious threat. If the shrinkage selects large $\lambda$ everywhere, the boundary-specific weights are unnecessary, and the contribution reduces to "ordinal stacking in conditional space." That result is still useful, but it is smaller.

Proposed Experiments

Data and metrics. We use WineQuality, Abalone, and Eucalyptus with the label binnings from the DK-NODE paper; the five AES-ENEM competencies with a BERTimbau CORN model as one base learner; and Cyst-X if access is granted. Metrics are mean absolute error, QWK, exact accuracy, ranked probability score, and expected calibration error, along with per-boundary weight dispersion and meta-learner train–test gap under subsampling.

Decisive checks. The first check is whether the full model beats the shared-weight ablation on any real dataset where weight dispersion is significant. The second is whether our calibration matches greedy selection while our accuracy matches nominal stacking, which would escape the calibration trap. The third is whether, at about 600 training points, shrinkage alone closes the overfitting gap that the IPMN study reported closing with augmentation (+0.022 QWK).

If all three checks fail, the conclusion is that ordinal stacking gains come from the base pool rather than the combiner.