Abstract
Variables in many massive high-dimensional data sets are structured, arising for example from measurements on a regular grid as in imaging and time series or from spatial-temporal measurements as in climate studies. Classical multivariate techniques ignore these structural relationships often resulting in poor performance. We propose a generalization of the singular value decomposition (SVD) and principal components analysis (PCA) that is appropriate for massive data sets with structured variables or known two-way dependencies. By finding the best low rank approximation of the data with respect to a transposable quadratic norm, our decomposition, entitled the Generalized least squares Matrix Decomposition (GMD), directly accounts for structural relationships. As many variables in high-dimensional settings are often irrelevant or noisy, we also regularize our matrix decomposition by adding two-way penalties to encourage sparsity or smoothness. We develop fast computational algorithms using our methods to perform generalized PCA (GPCA), sparse GPCA, and functional GPCA on massive data sets. Through simulations and a whole brain functional MRI example we demonstrate the utility of our methodology for dimension reduction, signal recovery, and feature selection with high-dimensional structured data.
Introduction
In the era of massive data collection, researchers frequently encounter high-dimensional datasets where variables possess inherent structural relationships. Examples include pixels in an image, voxels in a brain scan, or time-series measurements in climate science. Standard multivariate techniques, such as Principal Components Analysis (PCA) and Singular Value Decomposition (SVD), are the workhorses of dimension reduction. However, these methods treat every variable as independent and identically distributed, ignoring the spatial or temporal geometry of the data.
In structured data, noise is rarely random; it is often "structural." In functional MRI (fMRI), for instance, neighboring voxels may fluctuate together due to hardware noise or blood flow rather than neural activity. Standard PCA often captures this structural noise in its primary components, effectively burying the actual biological signal. This paper introduces the Generalized least squares Matrix Decomposition (GMD), a framework designed to bridge the gap between classical matrix decomposition and the complex, dependent nature of modern scientific data.
The GMD Framework
The core innovation of GMD is the shift from the standard Frobenius norm to a transposable quadratic norm. While a standard SVD seeks to minimize the squared differences between the data matrix $\mathbf{X}$ and its low-rank approximation, GMD minimizes the following objective:
$$|| \mathbf{X} - \mathbf{UDV}^T ||^2_{\mathbf{Q,R}}$$
Quadratic Operators
The matrices $\mathbf{Q}$ and $\mathbf{R}$ act as weights that encode known relationships within the data:
- $\mathbf{Q}$ (Row Dependencies): Handles relationships between observations, such as spatial proximity or graph-based connections.
- $\mathbf{R}$ (Column Dependencies): Handles relationships between variables, such as temporal smoothing or known feature correlations.
By utilizing Graph Laplacians as quadratic operators, GMD informs the decomposition algorithm about which variables are neighbors, forcing the resulting factors to prioritize signals that respect the data's underlying geometry.
Regularization and Computation
High-dimensional settings often contain irrelevant or noisy variables. To address this, the authors introduce Generalized Penalized Matrix Factorization (GPMF). This extension allows for the integration of various penalties directly into the GMD framework, such as the Lasso for sparsity or the $\Omega$-norm for smoothness. This unified approach enables the execution of Sparse GPCA and Functional GPCA within a single mathematical context.
Computational Efficiency
Processing "ultra high-dimensional" data requires significant computational resources. The authors developed a "power method" style algorithm for GMD that iteratively updates factors without requiring massive matrix inversions or full eigendecompositions. This ensures the method remains scalable to datasets with thousands of voxels or time points.
Experimental Results
The utility of GMD was evaluated through both synthetic simulations and real-world applications.
Simulations
In spatio-temporal simulations mimicking brain activation patterns (sinusoidal signals on a grid), standard PCA and Sparse PCA failed to isolate the regions of interest due to high levels of autoregressive noise. In contrast, GMD and Sparse GPCA recovered the spatial and temporal patterns with high fidelity, demonstrating superior signal recovery in low signal-to-noise ratio (SNR) environments.
Case Study: StarPlus fMRI
The methodology was applied to the StarPlus fMRI dataset, which tracks brain activity while subjects perform cognitive tasks. The results were striking:
- Variance Explanation: The first Generalized Principal Component (GPC) explained more variance than the first 20 standard PCs combined.
- Interpretability: The spatial maps produced by Sparse GPCA clearly highlighted the "ventral stream"—the brain's pathway for object identification. Standard PCA results, by comparison, were dominated by uninterpretable scanner noise.
Conclusion and Future Directions
GMD provides a robust alternative to traditional SVD/PCA for structured data, offering better signal recovery and more interpretable features. While the current framework assumes that the quadratic operators $\mathbf{Q}$ and $\mathbf{R}$ are known a priori (e.g., based on a grid layout), future work may focus on data-driven methods to estimate these operators simultaneously with the decomposition. Additionally, while the non-convex nature of the penalized version (GPMF) can lead to local optima, the use of unpenalized GMD solutions as starting points provides a reliable path toward meaningful biological and physical insights.
Summary updated