Learning generative models for valid knockoffs using novel multivariate-rank based statistics

Shoaib Bin Masud, Shuchin AeronarXiv 2021 · OpenReview

Abstract

The framework of optimal transport has been leveraged to extend the notion of rank to the multivariate setting while preserving desirable properties of the resulting goodness-of-fit (GoF) statistics. In particular, the rank energy (RE) and rank maximum mean discrepancy (RMMD) are distribution-free under the null, exhibit high power in statistical testing, and are robust to outliers. In this paper, we point to and alleviate some of the practical shortcomings of these proposed GoF statistics, namely their high computational cost, high statistical sample complexity, and lack of differentiability with respect to the data. We show that all these practically important issues are addressed by considering entropy-regularized optimal transport maps in place of the rank map, which we refer to as the soft rank. We consequently propose two new statistics, the soft rank energy (sRE) and soft rank maximum mean discrepancy (sRMMD), which exhibit several desirable properties. Given $n$ sample data points, we provide non-asymptotic convergence rates for the sample estimate of the entropic transport map to its population version that are essentially of the order $n^{-1/2}$ when the starting measure is subgaussian and the target measure has compact support. This result is novel compared to existing results which achieve a rate of $n^{-1}$ but crucially rely on both measures having compact support. We leverage this result to demonstrate fast convergence of sample sRE and sRMMD to their population version making them useful for high-dimensional GoF testing. Our statistics are differentiable and amenable to popular machine learning frameworks that rely on gradient methods. We leverage these properties towards showcasing the utility of the proposed statistics for generative modeling on two important problems: image generation and generating valid knockoffs for controlled feature selection.

Introduction

In one dimension, the concept of "rank" (the position of a data point in a sorted list) is a fundamental tool for non-parametric statistics. However, extending this to multivariate data is challenging because there is no natural way to "sort" vectors. Recent breakthroughs have utilized Optimal Transport (OT) to map complex multivariate distributions to a simple reference distribution (such as a uniform cube), thereby defining a multivariate rank. While statistics like Rank Energy (RE) and Rank Maximum Mean Discrepancy (RMMD) offer robustness and high power, they suffer from significant practical limitations: high computational cost, a sample complexity that degrades with dimensionality ($n^{-1/d}$), and a lack of differentiability that prevents their use as loss functions in modern gradient-based learning frameworks.

Methods: Soft Rank and Entropic Regularization

To address these challenges, we introduce Soft Rank, which replaces the standard OT map with an entropy-regularized version. By adding an entropy penalty to the OT problem, it can be solved much faster using the Sinkhorn algorithm. Crucially, this regularization makes the resulting statistics differentiable, allowing them to be integrated directly into neural network architectures.

Table of symbols and their meanings for multivariate rank-based statistics.

We propose two new statistics: soft rank energy (sRE) and soft rank maximum mean discrepancy (sRMMD). A key theoretical contribution of this work is proving that for subgaussian distributions, the entropic map converges at a rate of $n^{-1/2}$, effectively breaking the "curse of dimensionality" associated with unregularized OT. This fast convergence makes these statistics highly effective for high-dimensional goodness-of-fit testing and generative modeling.

Experimental Evaluation

We evaluate the proposed methods across three primary domains: generative modeling, knockoff generation, and real-world biomarker discovery.

Generative Modeling on MNIST

Models trained using sRE and sRMMD as loss functions within Generative Moment Matching Networks (GMMN) produced diverse and high-quality handwritten digits. In contrast, standard MMD-based models often suffered from "mode collapse," failing to capture the full variety of the dataset (e.g., predominantly producing a single digit class).

Valid Knockoffs for Feature Selection

Generating "knockoffs"—synthetic features that mimic the correlation structure of real features but are independent of the response—is critical for controlled feature selection. Our sRMMD-based approach successfully generated valid knockoffs for non-Gaussian and heavy-tailed data (e.g., Student’s t), significantly outperforming traditional second-order methods and deep learning benchmarks like DDLK in controlling the False Discovery Rate (FDR) while maintaining high detection power.

Metabolomics and Biomarker Discovery

In a study of Inflammatory Bowel Disease (IBD) involving 91 metabolites across 546 samples, sRMMD identified 16 out of 18 known significant biomarkers. It achieved a detection power of 76%, notably higher than standard MMD (63%) or traditional second-order methods (69%).

Conclusion and Future Work

By leveraging entropic regularization, we have developed multivariate rank statistics that are computationally efficient, statistically robust in high dimensions, and fully differentiable. These properties make sRE and sRMMD powerful tools for modern machine learning tasks. While the current work assumes a uniform target distribution on a cube, future research could explore alternative targets, such as spherical distributions, and extending theoretical guarantees to measures with non-compact support.

Summary updated

Abstract

The framework of optimal transport has been leveraged to extend the notion of rank to the multivariate setting while preserving desirable properties of the resulting goodness-of-fit (GoF) statistics. In particular, the rank energy (RE) and rank maximum mean discrepancy (RMMD) are distribution-free under the null, exhibit high power in statistical testing, and are robust to outliers. In this paper, we point to and alleviate some of the practical shortcomings of these proposed GoF statistics, namely their high computational cost, high statistical sample complexity, and lack of differentiability with respect to the data. We show that all these practically important issues are addressed by considering entropy-regularized optimal transport maps in place of the rank map, which we refer to as the soft rank. We consequently propose two new statistics, the soft rank energy (sRE) and soft rank maximum mean discrepancy (sRMMD), which exhibit several desirable properties. Given $n$ sample data points, we provide non-asymptotic convergence rates for the sample estimate of the entropic transport map to its population version that are essentially of the order $n^{-1/2}$ when the starting measure is subgaussian and the target measure has compact support. This result is novel compared to existing results which achieve a rate of $n^{-1}$ but crucially rely on both measures having compact support. We leverage this result to demonstrate fast convergence of sample sRE and sRMMD to their population version making them useful for high-dimensional GoF testing. Our statistics are differentiable and amenable to popular machine learning frameworks that rely on gradient methods. We leverage these properties towards showcasing the utility of the proposed statistics for generative modeling on two important problems: image generation and generating valid knockoffs for controlled feature selection.
BibTeX
@misc{masud2021learning,
  author = {Shoaib Bin Masud and Shuchin Aeron},
  title = {{Learning generative models for valid knockoffs using novel multivariate-rank based statistics}},
  year = {2021},
  eprint = {2111.00043},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.00043},
}
Figure