Abstract
Independent component analysis (ICA) is a fundamental statistical tool used to reveal hidden generative processes from observed data. However, traditional ICA approaches struggle with the rotational invariance inherent in Gaussian distributions, often necessitating the assumption of non-Gaussianity in the underlying sources. This may limit their applicability in broader contexts. To accommodate Gaussian sources, we develop an identifiability theory that relies on second-order statistics without imposing further preconditions on the distribution of sources, by introducing novel assumptions on the connective structure from sources to observed variables. Different from recent work that focuses on potentially restrictive connective structures, our proposed assumption of structural variability is both considerably less restrictive and provably necessary. Furthermore, we propose two estimation methods based on second-order statistics and sparsity constraint. Experimental results are provided to validate our identifiability theory and estimation methods.
Introduction
Independent Component Analysis (ICA) is a cornerstone of signal processing and machine learning, designed to recover latent sources from observed linear mixtures. A fundamental limitation of classical ICA, however, is the "non-Gaussianity requirement." Because the joint distribution of independent Gaussian variables is rotationally invariant—appearing as a perfect sphere in latent space—traditional methods cannot distinguish between the true source axes and any orthogonal rotation of them. This has historically forced researchers to assume sources are non-Gaussian or non-stationary.
This paper challenges this long-standing constraint. By shifting the focus from the distribution of the sources to the connective structure (sparsity) of the mixing matrix, the authors demonstrate that Gaussian sources are identifiable using only second-order statistics. This work bridges the gap between ICA and causal discovery, providing a more flexible framework for fields like biology and electronics where Gaussian signals (e.g., thermal noise) are prevalent.
Theoretical Framework
The core of the proposed theory lies in the realization that while Gaussian distributions are rotationally invariant, the sparsity of the mixing matrix $A$ is not. If the true mixing matrix is sparse, rotating it will almost certainly result in a denser matrix. This allows sparsity to act as a "tie-breaker" to recover the unique latent sources.
Structural Variability
The authors introduce Structural Variability (Assumption 1) as a necessary condition for identifiability. This condition requires that every pair of columns in the support of the mixing matrix must differ by more than one entry. In practical terms, this means no two sources should influence nearly identical sets of observed variables. The paper proves that if this condition is violated, there exists a rotation that preserves or increases sparsity, rendering the true sources unrecoverable.
The Causal Link
The framework further assumes that the mixing matrix can be permuted into a lower triangular form (Assumption 2). This connects ICA directly to Causal Discovery, where such a structure represents a Directed Acyclic Graph (DAG). By establishing this link, the authors show that identifying a sparse ICA model under these conditions is equivalent to finding a unique DAG, unifying two previously distinct areas of statistical learning.
Estimation Methods
To implement this theory, the authors propose two SparseICA estimation methods that utilize second-order statistics and sparsity constraints:
- Decomposition-based (SparseICA-D): This method directly minimizes the $L_0$ norm (the number of non-zero entries) of the mixing matrix while ensuring it matches the observed covariance of the data.
- Likelihood-based (SparseICA-L): This approach minimizes a Gaussian negative log-likelihood combined with a Minimax Concave Penalty (MCP) to encourage sparsity.
To handle the "lower triangular" requirement within a continuous optimization framework, the authors utilize a trace-based penalty function $g(A)$. This function equals zero only when the matrix satisfies the required DAG-like structure, allowing for efficient gradient-based optimization.
Experimental Validation
The researchers validated their theory using synthetic datasets consisting of 10 latent sources with varying ratios of Gaussian to non-Gaussian (exponential) distributions. The mixing matrices were generated to satisfy structural variability (e.g., polytrees), and performance was measured using the Mean Correlation Coefficient (MCC) and Amari distance.
Key Findings
- Robustness to Gaussianity: While standard algorithms like FastICA fail as the number of Gaussian sources increases, the proposed SparseICA methods maintain high accuracy (MCC $\approx$ 0.8) even when 100% of the sources are Gaussian.
- Sample Efficiency: The estimators showed consistent improvement as sample sizes increased, confirming the theoretical consistency of the approach.
- Assumption Strength: The experiments demonstrated that "Structural Variability" is a strictly weaker and more general assumption than those used in previous state-of-the-art Gaussian ICA research.
Conclusion and Future Work
This research provides a rigorous theoretical foundation for identifying Gaussian sources in ICA by leveraging the sparsity of the mixing process. By proving that Structural Variability is a necessary condition, the authors offer the most flexible framework to date for Gaussian-inclusive ICA.
Future research directions include extending these results to systems with feedback loops (relaxing the lower triangular constraint), developing formal finite-sample analysis, and scaling the algorithms to high-dimensional real-world data such as fMRI or genomic sequences.
Summary updated