Learning from High Dimensional Low Sample Size Biomedical Data

857 concepts755 papers2 proposals

In standard machine learning applications, "big data" typically means having a massive number of samples ($n$). However, in modern biomedicine and genomics, researchers routinely face the exact opposite scenario: "fat data." High-throughput sequencing, multi-omics, and mass spectrometry generate datasets with tens of thousands or even millions of molecular features ($p$), but clinical constraints mean these features are measured across only a few dozen or hundreds of patients ($n$).

As noted by George Alexandru Adam et al. and Tongxin Wang et al., this extreme $p \gg n$ imbalance creates a severe risk of overfitting. When a model possesses millions of trainable parameters but only a handful of examples, it can easily find an infinite number of complex mathematical surfaces that perfectly separate the training samples. Instead of learning genuine biological mechanisms, the model simply memorizes the stochastic noise unique to those specific patients, causing it to fail dramatically when deployed on independent cohorts.

To build models that generalize, researchers must restrict the model's degrees of freedom. Three core strategies have emerged to solve this problem: injecting biological structure, forcing information bottlenecks, and performing sparse feature selection.

Constraining the Search Space with Biological Priors

One of the most intuitive ways to prevent a model from learning arbitrary noise is to restrict its search space using what we already know about biology. Instead of using a fully connected "black box" neural network where any input feature can connect to any hidden unit, the network's architecture can be forced to mirror known biological networks, such as the human interactome or signaling pathways.

For instance, Aidong Zhang et al. introduced the Factor Graph Neural Network (FGNN), an architecture where hidden units explicitly represent biological entities like pathways rather than abstract mathematical constructs. Similarly, Jiarui Feng et al. developed DeepSigSurvNet, which forces the internal layers of a neural network to mimic biological signaling pathways. By structuring the network this way, the model is physically incapable of learning non-biological combinations of genes; it can only adjust weights along pathways known to exist in nature.

The authors of Zhichao Hou et al. expanded on this concept with PathExpSurv, which uses known gene-pathway relationships as a structural foundation but incorporates a specialized two-phase training scheme allowing the model to safely "expand" and discover missing biological links without overfitting.

This structural guidance can also be applied via graph-based regularization outside of deep learning. Anneleen Daemen et al. demonstrated that encoding the human interactome into machine learning kernels denoises cancer prognosis models by accounting for how genes interact in vivo, rather than treating them as isolated variables. Pietro Lio outlined a similar philosophy using network-based Cox regression, demonstrating that integrating a-priori gene-interaction networks allows survival models to identify robust, generalizable biomarkers instead of overfitting to single genes.

Information Bottlenecks via Feature Compression

Another strategy to combat the curse of dimensionality is to force the high-dimensional input through a tight information bottleneck before making a prediction. By using unsupervised or self-supervised learning, models can learn a compact, low-dimensional representation of the data that captures the true underlying biological variance while discarding random noise.

Autoencoders are highly effective for this task. Guocai Chen et al. showed that using a deep autoencoder to compress gene expression and pathway features significantly reduces the number of parameters a downstream model needs to learn, leading to much higher accuracy in drug combination synergy predictions.

For multi-omics integration, where combining different data types (like genomics and proteomics) causes the feature space to explode even further, Xiaoyu Zhang et al. developed OmiVAE. This framework combines a Variational Autoencoder (VAE) with a classification network to compress DNA methylation and gene expression data into a shared, low-dimensional latent space.

Single-author works have also demonstrated the power of this approach. Shekoofeh Azizi leveraged stacked autoencoders to extract robust regulatory signals from microRNA data, while Qingguo Zhou et al. applied convolutional autoencoders to reduce the dimensionality of noisy mass spectrometry data. By compressing the data first, the final classification network operates on a small number of robust latent features rather than thousands of noisy raw variables.

Sparse Selection and Parameter Thinning

Instead of compressing all features into a latent space, models can alternatively isolate a minimal, highly predictive subset of features or modify how parameters are allocated. Feature selection acts as a hard filter, stripping away the thousands of irrelevant dimensions that cause overfitting.

Early work by Per Lidén et al. explored rule induction to generate human-readable "if-then" logic that relies on only a handful of critical genes. To automate this process across complex datasets, Γεώργιος Παπουτσόγλου et al. utilized Automated Machine Learning (AutoML) to rapidly discover minimal-feature biosignatures that maintain out-of-sample accuracy while drastically reducing biomarker counts for clinical deployment. From an algorithmic standpoint, Claudio Gentile proposed multiplicative large margin algorithms that leverage multiplicative update rules to find small, highly accurate gene subsets from microarrays much faster than standard additive methods like traditional Support Vector Machines.

For deep learning models where feature selection must happen dynamically, Xiangjian Jiang et al. introduced ProtoGate, which uses a "global-to-local" feature selection gating mechanism combined with a non-parametric, prototype-based predictor to prevent overfitting on tabular biomedical data.

Finally, rather than reducing the features themselves, Adriana Romero et al. tackled the parameter explosion directly with "Diet Networks." Their architecture uses an auxiliary network to compute a lightweight, "thin" set of parameters for the input layer based on external feature representations (such as genomic positions). This ensures that the model's parameter count does not scale linearly with the massive number of genomic features, fundamentally altering the parameter-to-sample ratio.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 737 papers
  2. Version 1 · 2026-10-04 · 1282 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_curse_of_dimensionality_high_dimensional_51cf101c,
  title = {curse of dimensionality (high-dimensional low-sample-size problem)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/curse-of-dimensionality-high-dimensional-low-sample-size-problem/rdr_51cf101c55aa41f284df5caab186132a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_curse_of_dimensionality_high_dimensional_51cf101c_v2,
  title = {curse of dimensionality (high-dimensional low-sample-size problem)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/curse-of-dimensionality-high-dimensional-low-sample-size-problem/rdr_51cf101c55aa41f284df5caab186132a/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake