Spectral gaps and measure decompositions

March T. Boedihardjo, Joe Kileel, Vandy TombsarXiv 2026 · DOI · arXiv

TL;DR

This paper introduces the 4th moment operator as a computationally efficient tool for analyzing high-dimensional probability measures. By examining the "spectral gap" between its largest eigenvalues, the authors provide a way to detect whether a complex distribution can be decomposed into simpler subpopulations with distinct statistical signatures.

What Problem Does This Paper Address?

In high-dimensional statistics, understanding the underlying structure of a probability measure $\mu$ is notoriously difficult. Researchers often look for quantities that are "orthogonally invariant"—meaning they don't change if you rotate your data—to describe the shape of the distribution.

Commonly used metrics, such as the isoperimetric constant or the subgaussian constant, provide deep insights into a measure's geometry. However, there is a catch: calculating these values typically requires looking at the data from every possible direction. In high dimensions ($d$), this leads to an "exponential explosion" in computation time, making them practically useless for large datasets. Even working with the full 4th-order moment tensor is often NP-hard.

The authors set out to find a set of quantities that are both orthogonally invariant and computable in polynomial time, which can still tell us if a distribution is "hiding" a mixture of different subpopulations.

Methods

The core innovation is the 4th moment operator ($T_\mu$).

  1. Flattening the Tensor: Instead of struggling with a 4th-order tensor (a 4D array of numbers), the authors "flatten" it into a linear operator that acts on $d \times d$ symmetric matrices.
  2. Eigenvalue Analysis: They study the eigenvalues of this operator. Because the operator is symmetric and positive semidefinite, its eigenvalues are real and non-negative.
  3. Invariance: They prove that these eigenvalues are invariant under orthogonal transformations (rotations).
  4. $L^p-L^2$ Equivalence: To provide rigorous bounds, the authors rely on a common assumption in high-dimensional probability that relates higher-order moments to the second moment (covariance). This essentially controls the "heaviness" of the distribution's tails.

Datasets

As a theoretical mathematics and probability paper, this work does not use empirical "real-world" datasets. Instead, it uses several canonical synthetic distributions to demonstrate the theory:

  • Standard Normal Distributions: Used to show the baseline "spectral gap" behavior.
  • Mixtures of Gaussians: Used to demonstrate how overlapping subpopulations close the spectral gap.
  • Uniform Distributions: Evaluated on spheres, hypercubes, and discrete point sets to show how different geometries affect the operator's spectrum.

Key Insights and Ideas

  • The Spectral Gap Analogy: The authors draw a brilliant parallel to spectral graph theory. In a graph, a gap between the first and second eigenvalues of the adjacency matrix tells you if the graph is connected or made of separate clusters. Here, the gap between the first and second eigenvalues of $T_\mu$ tells you if the probability measure is a "single blob" or a mixture of statistically distinct components.
  • Outlier Eigenvalues: For simple, "well-behaved" distributions like a single Gaussian, the largest eigenvalue $\lambda_1$ is an outlier (scaling with the dimension $d$), while all other eigenvalues are small (scaling as constants).
  • Detecting Hidden Structure: If the second eigenvalue $\lambda_2$ is also large (i.e., $\lambda_1 \approx \lambda_2$), it is a "smoking gun" indicating that the measure can be decomposed into two parts with significantly different covariance structures.
  • Efficiency: Unlike many geometric constants, the eigenvalues of $T_\mu$ can be computed in polynomial time relative to the dimension $d$, making this approach scalable.

Key Results

  • Theorem 1.12: Establishes that the largest eigenvalue $\lambda_1$ is always significantly larger than the average eigenvalue (at least by a factor of $d/2$).
  • Theorem 1.17 (The Main Result): Provides a quantitative "sandwich" bound. It relates the second-order separation parameter $s(\mu)$—a measure of how different two sub-populations are—to the ratio of the second and first eigenvalues ($\lambda_2 / \lambda_1$).
  • Quantitative Examples:
    • For a standard normal distribution $\mathcal{N}(0, I)$, $\lambda_1 = d+2$ and $\lambda_2 = 2$. The gap is huge.
  • For a mixture of two normal distributions with different covariances, $\lambda_1 = \lambda_2 = d/4 + 1$. The gap vanishes, signaling the mixture.

Limitations and Future Work

  • Focus on Second-Order Statistics: The current framework is designed to detect differences in covariances (second-order statistics). While the authors suggest a way to handle differences in means (first-order statistics) by lifting the measure to $d+1$ dimensions, the primary strength is in covariance detection.
  • Moment Requirements: The strongest results (Theorem 1.17) require the existence of 8th moments ($\mu \in \mathcal{P}_8$), which might be restrictive for very heavy-tailed data.
  • Future Directions: The paper opens the door for new spectral clustering algorithms that use the 4th moment operator to partition high-dimensional data more effectively than standard methods that only look at the covariance matrix.

Summary updated

Abstract

Let $μ$ be a probability measure on $\mathbb{R}^{d}$. In this paper, we introduce a new set of computable quantities in $μ$ that are invariant under orthogonal transformations, namely, the eigenvalues of the 4th moment operator of $μ$. We show how the first and second largest eigenvalues of this operator can determine the extent to which $μ$ can be decomposed as an equal weight mixture of two probability measures with significantly different second order statistics.
BibTeX
@misc{boedihardjo2026spectral,
  author = {March T. Boedihardjo and Joe Kileel and Vandy Tombs},
  title = {{Spectral gaps and measure decompositions}},
  year = {2026},
  eprint = {2603.03245},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.03245},
}
Figure