Neyman-Pearson classification: parametrics and sample size requirement

Xin Tong, Lucy Xia, Jiacheng Wang, Yang FengarXiv 2018 · DOI · arXiv

Abstract

The Neyman-Pearson (NP) paradigm in binary classification seeks classifiers that minimize Type II error while strictly controlling the prioritized Type I error below a user-specified level $\alpha$. While recent nonparametric "umbrella algorithms" provide a distribution-free framework for this task, they impose a rigorous minimum sample size requirement on the prioritized class (Class 0). This requirement often renders NP classification unusable in data-scarce applications such as rare disease diagnosis. In this work, we introduce a parametric approach based on Linear Discriminant Analysis (LDA) that bypasses these sample size constraints. We propose four LDA-based NP classifiers designed for both low- and high-dimensional settings and introduce an adaptive sample splitting scheme to optimize the trade-off between model training and threshold estimation. Theoretical analysis and empirical results demonstrate that our parametric approach maintains error control even with small samples and achieves superior convergence rates for excess Type II error.

Introduction

In classical binary classification, the objective is typically to minimize the overall risk or misclassification error. However, in many high-stakes domains—such as medical diagnosis, spam detection, or anomaly detection—the costs associated with different types of errors are highly asymmetric. For instance, in cancer screening, a Type I error (a false negative, where a sick patient is classified as healthy) is far more critical than a Type II error (a false positive, leading to unnecessary further testing).

The Neyman-Pearson (NP) paradigm addresses this asymmetry by treating the Type I error as a hard constraint. The goal is to find a classifier that minimizes the Type II error subject to the constraint that the Type I error remains below a threshold $\alpha$ with high probability. While nonparametric methods have recently gained popularity for their flexibility, their reliance on order statistics creates a "sample size floor." If the number of Class 0 observations is too small, these methods cannot statistically guarantee the $\alpha$ bound, leaving practitioners without a viable model.

Moving from Nonparametric to Parametric NP Classification

To address the limitations of nonparametric methods, we leverage the parametric assumptions of Linear Discriminant Analysis (LDA). By assuming that the features follow a multivariate Gaussian distribution with a common covariance matrix, we can derive a parametric thresholding rule ($C_\alpha^p$) that does not depend on a minimum sample size of Class 0.

The Four Proposed Classifiers

We developed four distinct NP classifiers to handle various data environments:

  1. NP-LDA: Uses LDA for scoring but retains a nonparametric thresholding rule.
  2. NP-sLDA: A sparse version of NP-LDA using a Lasso-type penalty for high-dimensional data ($d \gg n$).
  3. pNP-LDA: Uses LDA for both scoring and the new parametric thresholding rule.
  4. pNP-sLDA: The sparse version of the fully parametric approach, ideal for high-dimensional, small-sample scenarios.

Adaptive Sample Splitting

NP classification typically involves splitting Class 0 data: one portion to train the scoring function and the other to determine the threshold. Traditionally, this split is fixed (e.g., 50/50). We propose a new adaptive sample splitting scheme that uses a data-driven approach to select the optimal split ratio $\tau$. This strategy ensures that the available data is used efficiently, significantly reducing the Type II error without compromising the Type I error constraint.

Experimental Evaluation

Simulation Studies

We tested our methods against various covariance structures (AR(1) and Compound Symmetry). In high-dimensional simulations, the sparse variants (sLDA) proved significantly more robust. Standard LDA often produced overly conservative thresholds as the number of features increased, whereas sLDA maintained high power (low Type II error).

Real-World Applications

We applied our classifiers to two genomic datasets:

  • Neuroblastoma Dataset: A high-dimensional dataset ($d = 43,827$) with 498 samples. The NP-sLDA and pNP-sLDA methods successfully identified high-risk patients while keeping false negatives strictly controlled.
  • Breast Cancer Dataset: This dataset ($d = 22,215$) featured only 118 samples. When the error bound $\alpha$ was set to 0.05, the existing nonparametric umbrella algorithm failed to return a result ("Unable to Process") because it lacked the required 59 Class 0 samples. In contrast, our pNP-sLDA classifier successfully processed the data, providing a functional model that respected the $\alpha$ constraint.

Key Insights

  • The Small Sample Advantage: Parametric assumptions serve as a statistical bridge. When data is too scarce to support nonparametric ranking, the Gaussian assumption allows for the calculation of a valid threshold, making NP classification possible for rare diseases.
  • Theoretical Efficiency: We proved that under the LDA model assumption, our parametric NP classifiers achieve a convergence rate for excess Type II error of $n^{-1/4}$, which is theoretically superior to nonparametric alternatives.
  • Optimizing the Split: The adaptive splitting scheme demonstrates that the "optimal" way to divide data depends on the specific dataset. By finding the sweet spot between scoring accuracy and threshold precision, we can maximize the power of the classifier.

Conclusion and Future Work

By integrating parametric LDA models into the Neyman-Pearson paradigm, we have expanded the reach of NP classification to small-sample and high-dimensional regimes. This approach is particularly valuable for medical research where Class 0 (the prioritized class) is often the most difficult to collect.

Future research will explore extending these parametric rules to Quadratic Discriminant Analysis (QDA) to accommodate different class covariances and investigating heavy-tailed distributions to improve robustness in financial or non-Gaussian genomic data.

Summary updated

Abstract

The Neyman-Pearson (NP) paradigm in binary classification seeks classifiers that achieve a minimal type II error while enforcing the prioritized type I error controlled under some user-specified level $α$. This paradigm serves naturally in applications such as severe disease diagnosis and spam detection, where people have clear priorities among the two error types. Recently, Tong, Feng and Li (2018) proposed a nonparametric umbrella algorithm that adapts all scoring-type classification methods (e.g., logistic regression, support vector machines, random forest) to respect the given type I error upper bound $α$ with high probability, without specific distributional assumptions on the features and the responses. Universal the umbrella algorithm is, it demands an explicit minimum sample size requirement on class $0$, which is often the more scarce class, such as in rare disease diagnosis applications. In this work, we employ the parametric linear discriminant analysis (LDA) model and propose a new parametric thresholding algorithm, which does not need the minimum sample size requirements on class $0$ observations and thus is suitable for small sample applications such as rare disease diagnosis. Leveraging both the existing nonparametric and the newly proposed parametric thresholding rules, we propose four LDA-based NP classifiers, for both low- and high-dimensional settings. On the theoretical front, we prove NP oracle inequalities for one proposed classifier, where the rate for excess type II error benefits from the explicit parametric model assumption. Furthermore, as NP classifiers involve a sample splitting step of class $0$ observations, we construct a new adaptive sample splitting scheme that can be applied universally to NP classifiers, and this adaptive strategy reduces the type II error of these classifiers.
BibTeX
@misc{tong2018neyman,
  author = {Xin Tong and Lucy Xia and Jiacheng Wang and Yang Feng},
  title = {{Neyman-Pearson classification: parametrics and sample size requirement}},
  year = {2018},
  eprint = {1802.02557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.02557},
}
Figure