Zero-Complement Extrapolation: Parameter Recovery and Classification in Severely Imbalanced Logistic Regression under Rank-Deficient Regimes

Chongsun Park

TL;DR

When the minority class in a dataset is extremely rare—fewer samples than there are features—standard logistic regression and oversampling techniques like SMOTE fail because they "collapse" into a low-dimensional subspace. This paper introduces Zero-Complement Extrapolation, a method that recovers the true decision boundary by analyzing the "tilt" in the abundant majority class and extrapolating synthetic minority points outward from the majority's edge.

What Problem Does This Paper Address?

In fields like rare disease diagnosis or fraud detection, we often face "severely imbalanced" data. The authors focus on a particularly nasty version called rank-deficient rarity, where the number of minority instances ($n_1$) is smaller than the number of features ($p$).

In this regime, three things go wrong:

  1. Maximum Likelihood Estimation (MLE) fails: The model experiences "complete separation," where the weights blow up to infinity.
  2. Penalized methods (like Firth’s regression) struggle: While they stop the weights from exploding, they introduce a massive bias in the intercept, often leading to zero sensitivity (the model never predicts the rare class).
  3. SMOTE collapses: Popular oversampling methods like SMOTE create new points by interpolating between existing minority points. If you only have 3 points in a 10D space, any interpolation is trapped on a flat 2D triangle. This "subspace collapse" leaves the model with zero variance in most directions, leading back to parameter explosion.

Methods

The authors propose a three-stage Self-Calibrated Zero-Complement estimator that flips the script: instead of trying to learn from the scarce minority, it learns from the "probabilistic footprint" left in the abundant majority.

  • Stage 1: Hyperplane Identification via Stein’s Lemma. The authors prove that if the data is normally distributed, the "drift" in the majority class mean (compared to the overall population mean) points exactly in the direction of the true decision boundary. They also use the intrinsic skewness of the majority class to help find this direction when the population mean is unknown.
  • Stage 2: Density Fringe Extrapolation. Instead of interpolating between noisy minority points, the algorithm identifies the "density cliff" or fringe of the majority class. It then "pushes" synthetic points outward from this fringe along a conjugate direction. This ensures the synthetic data has a "full-rank" covariance matrix, preventing the subspace collapse seen in SMOTE.
  • Stage 3: Zero-Boundary Calibration. Finally, the intercept is calibrated by anchoring it between the outer edge of the majority support and the inner edge of the minority samples, ensuring stable predictions at the standard 0.5 probability threshold.
Geometric illustration of subspace collapse in SMOTE vs. the full-rank preservation of Zero-Complement extrapolation.

Datasets

The method was evaluated using:

  • Monte Carlo Simulations: Controlled environments with $p=10$ and $n_1=5$ to test parameter recovery.
  • Wisconsin Diagnostic Breast Cancer (WDBC): A real-world dataset where the authors deliberately discarded 99% of malignant cases to create a $p=10, n_1=5$ scenario.
  • Banknote Authentication: A dataset where they enforced an extreme $p=4, n_1=3$ regime.

Key Insights and Ideas

  • The Majority "Remembers": Even if you delete almost all minority samples, the majority class distribution is "truncated" by the logistic function. This truncation leaves a measurable mark on the majority's mean and skewness that reveals the hidden boundary.
  • Extrapolation > Interpolation: In high-dimensional spaces with tiny samples, "filling in the gaps" (interpolation) is mathematically insufficient. You must "expand the territory" (extrapolation) to maintain the statistical regularity needed for stable model fitting.
  • Subspace Collapse is a Trap: The paper formally proves that SMOTE-like methods are guaranteed to fail when $n_1 < p$ because they produce a rank-deficient covariance matrix, which is the root cause of numerical instability in logistic models.

Key Results

The proposed method significantly outperformed state-of-the-art baselines in extreme rarity conditions:

  • Parameter Recovery: In simulations, the "Regime A" version of the estimator recovered the true hyperplane with a cosine similarity of 0.9976, nearly matching the "Oracle" model that had access to the full dataset.
  • F1-Score Boost: On the Breast Cancer dataset ($n_1=5$), the Zero-Complement method achieved an F1-score of 0.8811, compared to just 0.6797 for Firth’s regression and 0.7748 for SMOTE.
  • Stability: The method showed much lower variance across different random subsamples, meaning it is far more reliable when you only have one small "lucky" (or unlucky) batch of minority data.

Limitations and Future Work

  • Gaussian Assumption: The theoretical proof relies on Stein’s Lemma, which assumes multivariate normal covariates. While empirical tests on non-Gaussian data (like the Banknote dataset) showed robustness, formal extensions to other distributions (the exponential family) are still needed.
  • Ultra-High Dimensions: The current method uses a ridge-regularized covariance matrix. For cases where the number of features $p$ is even larger than the majority sample size $n_0$, the authors suggest incorporating sparse Graphical Lasso techniques.
  • Non-linear Boundaries: The current framework is designed for linear logistic regression; extending this "fringe extrapolation" logic to kernel methods or neural networks remains an open challenge.

Summary updated

Abstract

Parameter estimation and classification in binary logistic regression deteriorate severely when the number of minority class observations ($n_1$) is smaller than the feature dimension ($p$). Under such rank-deficient rarity ($n_1 < p$), maximum likelihood estimators suffer from quasi-complete separation, penalized likelihood methods exhibit severe intercept bias, and convex minority oversampling techniques such as SMOTE collapse into an $(n_1 - 1)$-dimensional affine subspace. In this paper, we propose a novel \textit{Self-Calibrated Zero-Complement} estimator that shifts the paradigm from minority interpolation to majority fringe extrapolation. Grounded in Stein's Lemma, we prove that the conditional expectation shift of the abundant majority class ($Y=0$) induced by logistic truncation uniquely identifies the true hyperplane normal $\boldsymbolβ$ after Mahalanobis whitening. We further establish that extrapolating synthetic minority instances outward from the high-quantile density fringe of the majority distribution along the conjugate direction preserves full-rank covariance regularity ($\mathrm{rank} = p$). To accommodate both semi-supervised environments (with an unlabeled background mean) and strictly supervised subsamples, we introduce a hybrid estimator incorporating the intrinsic third-moment skewness of the majority class alongside a location-shift-invariant zero-boundary intercept calibration. Extensive Monte Carlo simulations and real-world benchmark applications (Wisconsin Diagnostic Breast Cancer and Banknote Authentication) under extreme rarity ($n_1 \in \{3, 5\}$) demonstrate that our approach prevents parameter divergence, recovers the oracle hyperplane with high fidelity, and substantially outperforms Firth's regression and SMOTE in F1-score and estimation stability.
BibTeX
@misc{park2026zero,
  author = {Chongsun Park},
  title = {{Zero-Complement Extrapolation: Parameter Recovery and Classification in Severely Imbalanced Logistic Regression under Rank-Deficient Regimes}},
  year = {2026},
  eprint = {2610.39194},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39194},
}
Figure