Abstract
Tractable generalizations of the Gaussian distribution play an important role for the analysis of high-dimensional data. One very general super-class of Normal distributions is the class of $\nu$-spherical distributions whose random variables can be represented as the product $\mathbf{x} = r \cdot \mathbf{u}$ of a uniformly distributed random variable $\mathbf{u}$ on the $1$-level set of a positively homogeneous function $\nu$ and an arbitrary positive radial random variable $r$. In this paper, we introduce a new subclass of $\nu$-spherical distributions by choosing $\nu$ to be a nested cascade of $L_p$-norms. This class is computationally tractable, including explicit expressions for the normalization constant and fast $O(n)$ sampling algorithms. We investigate its marginals, maximum likelihood fitting, and discuss its tight links to Independent Component Analysis (ICA) and Independent Subspace Analysis (ISA). Finally, we introduce the Nested Radial Factorization algorithm (NRF), a form of non-linear ICA.
1. Introduction
In high-dimensional data analysis, the multivariate Gaussian is the standard model due to its simplicity and analytical tractability. However, real-world data—such as natural images or audio—often exhibit heavy tails and complex dependencies that a Gaussian cannot capture. While broader classes like $\nu$-spherical distributions have been proposed, they are often computationally intractable because their normalization constants or sampling algorithms are unknown for arbitrary $\nu$.
We bridge this gap by defining $L_p$-nested symmetric distributions. This family is flexible enough to model non-Gaussian structures while remaining structured enough to allow for exact fitting, sampling, and normalization.
2. Methodology: Nested $L_p$-Norm Cascades
The core innovation is the hierarchical grouping of variables using a tree structure. Instead of a single norm defining the distribution's shape, we use a nested cascade:
- Hierarchical Grouping: Variables are grouped at the leaves of a tree.
- Nested Norms: Each internal node computes an $L_p$-norm of its children, allowing for different $p$ values at each node.
This structure allows for a variety of iso-density contours—the shapes where probability density is constant. By varying $p$ values, the model can represent distributions ranging from "spiky" (low $p$) to "boxy" (high $p$).
3. Key Insights and Theoretical Properties
- Statistical Dependencies: A key theoretical result is that the only distribution in this class with independent marginals is the standard $L_p$-spherically symmetric one. This implies that any non-trivial nested structure automatically models statistical dependencies between variables.
- Unified Framework: The framework encompasses and generalizes Independent Component Analysis (ICA) and Independent Subspace Analysis (ISA).
- Non-linear ICA (NRF): We introduce Nested Radial Factorization (NRF), which can "un-nest" signals with complex hierarchical dependencies into independent components.
4. Key Results
- Exact Normalization: We derived the partition function for an arbitrary $L_p$-nested tree, enabling direct likelihood-based model comparison.
- Efficient Sampling: We provide an exact sampling algorithm that runs in $O(n)$ time, making the model practical for high-dimensional applications.
- Robust Inference: We prove that Bayesian inference for the location parameter is robust to the choice of the radial distribution, extending known results for simpler spherical distributions.
5. Limitations and Future Work
While powerful, the current framework requires the tree structure (the grouping of variables) to be provided via domain knowledge or heuristics, as there is not yet an efficient way to learn the optimal tree directly from data. Additionally, jointly optimizing the whitening matrix and distribution parameters remains a non-convex challenge. Future work will explore these distributions as priors in over-complete linear models.
Summary updated