$L_p$-nested symmetric distributions

Fabian H. Sinz, Matthias Bethge

Abstract

Tractable generalizations of the Gaussian distribution play an important role for the analysis of high-dimensional data. One very general super-class of Normal distributions is the class of $\nu$-spherical distributions whose random variables can be represented as the product $\mathbf{x} = r \cdot \mathbf{u}$ of a uniformly distributed random variable $\mathbf{u}$ on the $1$-level set of a positively homogeneous function $\nu$ and an arbitrary positive radial random variable $r$. In this paper, we introduce a new subclass of $\nu$-spherical distributions by choosing $\nu$ to be a nested cascade of $L_p$-norms. This class is computationally tractable, including explicit expressions for the normalization constant and fast $O(n)$ sampling algorithms. We investigate its marginals, maximum likelihood fitting, and discuss its tight links to Independent Component Analysis (ICA) and Independent Subspace Analysis (ISA). Finally, we introduce the Nested Radial Factorization algorithm (NRF), a form of non-linear ICA.

1. Introduction

In high-dimensional data analysis, the multivariate Gaussian is the standard model due to its simplicity and analytical tractability. However, real-world data—such as natural images or audio—often exhibit heavy tails and complex dependencies that a Gaussian cannot capture. While broader classes like $\nu$-spherical distributions have been proposed, they are often computationally intractable because their normalization constants or sampling algorithms are unknown for arbitrary $\nu$.

We bridge this gap by defining $L_p$-nested symmetric distributions. This family is flexible enough to model non-Gaussian structures while remaining structured enough to allow for exact fitting, sampling, and normalization.

2. Methodology: Nested $L_p$-Norm Cascades

The core innovation is the hierarchical grouping of variables using a tree structure. Instead of a single norm defining the distribution's shape, we use a nested cascade:

  1. Hierarchical Grouping: Variables are grouped at the leaves of a tree.
  2. Nested Norms: Each internal node computes an $L_p$-norm of its children, allowing for different $p$ values at each node.
Figure 1: The nested function can be visualized as a tree where internal nodes compute $L_p$-norms.

This structure allows for a variety of iso-density contours—the shapes where probability density is constant. By varying $p$ values, the model can represent distributions ranging from "spiky" (low $p$) to "boxy" (high $p$).

Figure 2: An example of an iso-density contour with $p_1 = 10$, showing the model's capacity for peaked, non-Gaussian structures.

3. Key Insights and Theoretical Properties

  • Statistical Dependencies: A key theoretical result is that the only distribution in this class with independent marginals is the standard $L_p$-spherically symmetric one. This implies that any non-trivial nested structure automatically models statistical dependencies between variables.
  • Unified Framework: The framework encompasses and generalizes Independent Component Analysis (ICA) and Independent Subspace Analysis (ISA).
  • Non-linear ICA (NRF): We introduce Nested Radial Factorization (NRF), which can "un-nest" signals with complex hierarchical dependencies into independent components.

4. Key Results

  • Exact Normalization: We derived the partition function for an arbitrary $L_p$-nested tree, enabling direct likelihood-based model comparison.
  • Efficient Sampling: We provide an exact sampling algorithm that runs in $O(n)$ time, making the model practical for high-dimensional applications.
  • Robust Inference: We prove that Bayesian inference for the location parameter is robust to the choice of the radial distribution, extending known results for simpler spherical distributions.

5. Limitations and Future Work

While powerful, the current framework requires the tree structure (the grouping of variables) to be provided via domain knowledge or heuristics, as there is not yet an efficient way to learn the optimal tree directly from data. Additionally, jointly optimizing the whitening matrix and distribution parameters remains a non-convex challenge. Future work will explore these distributions as priors in over-complete linear models.

Summary updated

Abstract

Tractable generalizations of the Gaussian distribution play an important role for the analysis of high-dimensional data. One very general super-class of Normal distributions is the class of $ν$-spherical distributions whose random variables can be represented as the product $\x = r\cdot \u$ of a uniformly distribution random variable $\u$ on the $1$-level set of a positively homogeneous function $ν$ and arbitrary positive radial random variable $r$. Prominent subclasses of $ν$-spherical distributions are spherically symmetric distributions ($ν(\x)=\|\x\|_2$) which have been further generalized to the class of $L_p$-spherically symmetric distributions ($ν(\x)=\|\x\|_p$). Both of these classes contain the Gaussian as a special case. In general, however, $ν$-spherical distributions are computationally intractable since, for instance, the normalization constant or fast sampling algorithms are unknown for an arbitrary $ν$. In this paper we introduce a new subclass of $ν$-spherical distributions by choosing $ν$ to be a nested cascade of $L_p$-norms. This class is still computationally tractable, but includes all the aforementioned subclasses as a special case. We derive a general expression for $L_p$-nested symmetric distributions as well as the uniform distribution on the $L_p$-nested unit sphere, including an explicit expression for the normalization constant. We state several general properties of $L_p$-nested symmetric distributions, investigate its marginals, maximum likelihood fitting and discuss its tight links to well known machine learning methods such as Independent Component Analysis (ICA), Independent Subspace Analysis (ISA) and mixed norm regularizers. Finally, we derive a fast and exact sampling algorithm for arbitrary $L_p$-nested symmetric distributions, and introduce the Nested Radial Factorization algorithm (NRF), which is a form of non-linear ICA.
BibTeX
@article{sinz2010l,
  author = {Fabian H. Sinz and Matthias Bethge},
  title = {{$L\_p$-nested symmetric distributions}},
  journal = {J. Mach. Learn. Res.},
  year = {2010},
}
Figure