Deep Neural Networks for High Dimension, Low Sample Size Data

Bo Liu, Ying Wei, Yu Zhang, Qiang Yang

Abstract: Deep Neural Pursuit (DNP) is a novel framework designed for High Dimension, Low Sample Size (HDLSS) environments. By combining greedy incremental feature selection with a multiple dropout gradient averaging technique, DNP enables Deep Neural Networks to capture complex non-linear relationships without the massive overfitting typically associated with small datasets.

Introduction

In bioinformatics and other specialized fields, researchers frequently encounter the "curse of dimensionality": datasets with millions of features but only a few dozen or hundred samples. This High Dimension, Low Sample Size (HDLSS) problem makes standard Deep Neural Networks (DNNs) prone to extreme overfitting and unstable training due to high-variance gradients. While linear models like Lasso are stable, they often fail to capture the complex, hierarchical non-linear interactions inherent in biological systems.

The Deep Neural Pursuit (DNP) Framework

DNP addresses these challenges by bridging the gap between classical feature selection and deep learning. The framework employs two primary innovations:

  1. Greedy Incremental Selection: Rather than training on all features simultaneously, DNP starts with an empty input layer and adds features one by one. At each step, it selects the candidate feature with the largest gradient magnitude, indicating the greatest potential for loss reduction.
  2. Multiple Dropout Averaging: To stabilize selection in low-sample regimes, DNP performs multiple forward/backward passes with different dropout masks. Averaging these gradients reduces variance, ensuring that the selected features represent true signals rather than random noise.
The DNP framework: (1) Initializing candidate weights to zero, (2) Selecting features via gradient magnitude, and (3) Using multiple dropout to average gradients.

Experimental Results

The authors evaluated DNP against several baselines, including Lasso, HSIC-Lasso, and Gradient Boosted Feature Selection (GBFS), using both synthetic and real-world biological datasets (e.g., Colon, Leukemia, and GLI_85).

Synthetic Data and Non-Linearity

On synthetic data designed with non-linear ground truths, DNP achieved a feature selection F1 score near 1.0. Visual analysis of decision boundaries confirms that DNP accurately captures non-linear patterns that linear models like Logistic Regression ($l_1$) fail to identify.

Comparison of decision boundaries for various methods. DNP (c) effectively recovers the non-linear structure of the test samples (a) compared to linear (d) and other feature selection baselines (e, f).

Real-World Biological Performance

Across six biological datasets, DNP outperformed the best baseline (HSIC-Lasso) by an average of 2.53% in AUC. Notably, DNP reached peak performance using fewer than 10 features on most datasets, demonstrating the sparsity and complexity of biological signals. The "Multiple Dropout" version of DNP also showed significantly higher stability scores compared to versions without it, proving that stability is critical for reliable feature selection in small samples.

Conclusion

Deep Neural Pursuit demonstrates that the power of deep non-linear architectures can be harnessed for small datasets through controlled feature entry and gradient stabilization. By utilizing multiple dropout averaging, DNP provides a robust solution for feature selection in high-dimensional spaces, offering a significant advancement for genomic research and other HDLSS domains.

Summary updated

Abstract

Deep neural networks (DNN) have achieved breakthroughs in applications with large sample size. However, when facing high dimension, low sample size (HDLSS) data, such as the phenotype prediction problem using genetic data in bioinformatics, DNN suffers from overfitting and high-variance gradients. In this paper, we propose a DNN model tailored for the HDLSS data, named Deep Neural Pursuit (DNP). DNP selects a subset of high dimensional features for the alleviation of overfitting and takes the average over multiple dropouts to calculate gradients with low variance. As the first DNN method applied on the HDLSS data, DNP enjoys the advantages of the high nonlinearity, the robustness to high dimensionality, the capability of learning from a small number of samples, the stability in feature selection, and the end-to-end training. We demonstrate these advantages of DNP via empirical results on both synthetic and real-world biological datasets.
BibTeX
@inproceedings{liu2017deep,
  author = {Bo Liu and Ying Wei and Yu Zhang and Qiang Yang},
  title = {{Deep Neural Networks for High Dimension, Low Sample Size Data}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/318},
}
Figure