Self-Supervision Enhanced Feature Selection with Correlated Gates

Changhee Lee, Fergus Imrie, Mihaela van der SchaarICLR 2022 · OpenReview

In fields like medicine and genomics, researchers often face a "needle in a haystack" problem. They might have data on tens of thousands of genes (features) for a patient, but only a handful of those genes actually drive a specific disease. To make matters worse, labeled data—patients with confirmed diagnoses—is often scarce, and many genes are highly correlated. This makes it difficult to tell which features are truly important and which are just "along for the ride."

Presented at ICLR 2022, SEFS (Self-supervision Enhanced Feature Selection) offers a clever deep learning solution to these twin challenges: data scarcity and feature redundancy.

TL;DR

SEFS is a novel deep learning framework that performs feature selection by leveraging unlabeled data through self-supervised pre-training. By using a "correlated gating" mechanism based on Gaussian copulas, it forces redundant features to compete, allowing the model to identify the most relevant variables even when labeled samples are extremely limited.

What Problem Does This Paper Address?

Feature selection is the process of identifying the most informative subset of input variables for a predictive task. While deep learning has made strides here, it hits two major roadblocks in high-dimensional biological and clinical data:

  1. The Small-$n$-Large-$p$ Problem: When you have thousands of features ($p$) but only dozens of labeled samples ($n$), deep models tend to overfit, "memorizing" noise rather than learning true biological signals.
  2. Multicollinearity: Features in nature are rarely independent. If five genes are highly correlated, standard methods might pick all five (redundancy) or pick the wrong one entirely because they cannot distinguish their individual effects.

Existing methods usually assume features are independent during the selection process, which fails to account for the underlying structure of the data.

Methods

The authors introduce SEFS, which operates in two distinct phases to tackle these challenges.

SEFS Architecture Overview

1. The Self-Supervision Phase (Pre-training)

Before looking at any labels, the model trains on a large pool of unlabeled data. It solves two "pretext" tasks:

  • Feature Reconstruction: Given a random subset of features, can the model predict the values of the missing features?
  • Gate Vector Estimation: Can the model identify which features were masked and which were provided?

This phase "pre-conditions" the encoder to understand the data's internal structure and relationships, preventing it from overfitting once labels are introduced.

2. The Supervision Phase (Fine-tuning)

The pre-trained model is then fine-tuned on the small labeled dataset. Here, the model learns a selection probability ($\pi$) for each feature to optimize the final prediction task.

The Secret Sauce: Correlated Gates

The most significant innovation is how SEFS handles feature selection. Instead of treating each feature's "gate" (on/off switch) as an independent coin flip, SEFS uses a Multivariate Bernoulli distribution implemented via a Gaussian Copula.

By pre-calculating the correlation matrix of the input features, the model ensures that if two features are highly correlated, their "gates" are likely to be activated or deactivated together during training. This forces correlated features to compete against each other, eventually allowing the model to settle on the most "discriminative" one rather than selecting a redundant group.

Key Insights and Ideas

  • Self-Supervision as a Regularizer: The paper demonstrates that learning the "local structure" of data through reconstruction is a powerful way to combat the overfitting typically seen in high-dimensional, low-sample-size regimes.
  • Competition via Correlation: By explicitly modeling feature dependencies, the authors turn a weakness (multicollinearity) into a strength. The correlated gating mechanism acts as a filter that naturally suppresses redundant information.
  • Differentiable Selection: Using a continuous relaxation of the Multivariate Bernoulli distribution allows the entire feature selection process to be trained using standard gradient descent, making it compatible with modern deep learning pipelines.

Key Results

SEFS consistently outperformed all benchmarks, particularly when labeled data was most scarce.

Performance with Limited Labeled Data
  • Predictive Accuracy: On the UKCF clinical dataset (245 variables predicting respiratory failure), SEFS achieved an AUROC of 0.846, significantly higher than the next best supervised method (STG at 0.781).
  • High-Dimensional Robustness: In the PBMC transcriptomics dataset (over 19,000 features), SEFS maintained an AUROC of 0.884, while traditional methods like LASSO lagged at 0.703.
  • Biological Relevance: Qualitative analysis showed that SEFS identified features (like PI Alleles in Cystic Fibrosis or the S100A4 gene in T-cells) that are well-documented in medical literature, whereas other models often selected noisy or irrelevant neighbors.

Limitations and Future Work

  • Computational Complexity: Generating correlated gate vectors involves Cholesky decomposition of a correlation matrix. For datasets with tens of thousands of features, this can be slow. The authors mitigated this using clustering and thresholding, but more efficient sampling remains an area for growth.
  • Static Correlations: The model currently uses a pre-computed, fixed correlation matrix. A promising future direction would be to allow the model to learn and update these correlations dynamically during the training process.
  • Domain Specificity: While tested on omics and clinical data, the "correlated gates" concept could likely be extended to other structured data types, such as sensor networks or financial time series.

Summary updated

Abstract

Discovering relevant input features for predicting a target variable is a key scientific question. However, in many domains, such as medicine and biology, feature selection is confounded by a scarcity of labeled samples coupled with significant correlations among features. In this paper, we propose a novel deep learning approach to feature selection that addresses both challenges simultaneously. First, we pre-train the network using unlabeled samples within a self-supervised learning framework by solving pretext tasks that require the network to learn informative representations from partial feature sets. Then, we fine-tune the pre-trained network to discover relevant features using labeled samples. During both training phases, we explicitly account for the correlation structure of the input features by generating correlated gate vectors from a multivariate Bernoulli distribution. Experiments on multiple real-world datasets including clinical and omics demonstrate that our model discovers relevant features that provide superior prediction performance compared to the state-of-the-art benchmarks in practical scenarios where there is often limited labeled data and high correlations among features.
BibTeX
@inproceedings{lee2022self,
  author = {Changhee Lee and Fergus Imrie and Mihaela van der Schaar},
  title = {{Self-Supervision Enhanced Feature Selection with Correlated Gates}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=TRHU_xYiyux},
}
Figure