When decomposing high-dimensional data into latent factors, classical linear algebra relies heavily on a single geometric convenience: orthogonality. Whether performing Principal Component Analysis (PCA), Independent Component Analysis (ICA), or higher-order tensor decompositions like Tucker or Canonical Polyadic (CP) models, enforcing that basis vectors are perfectly perpendicular (at 90-degree angles) makes the underlying optimization highly tractable.
However, real-world data rarely conforms to this rigid geometric assumption. In fields ranging from neuroimaging to hyperspectral processing, the true underlying processes are often correlated, overlapping, or overcomplete. Forcing these systems into an orthogonal framework introduces severe model mismatch, leading to phantom components and poor reconstruction.
A major research direction focuses on relaxing or entirely removing these orthogonality constraints, developing algorithms and theoretical guarantees capable of recovering true, non-orthogonal latent structures directly from complex data.
The Failure of Orthogonal Deflation
To understand why non-orthogonal recovery requires fundamentally new mathematics, consider how standard decomposition algorithms work. The most common paradigm is deflation: the algorithm finds the single most dominant component, projects the data onto the orthogonal complement of that component to "subtract" its variance, and repeats the process to find subsequent components.
As demonstrated by Lester Mackey et al. in Deflation Methods for Sparse PCA, this greedy subtraction is mathematically unsound when components are non-orthogonal or sparse. When the true latent factors are correlated, projecting the data onto a perpendicular subspace to eliminate the first component inadvertently corrupts the remaining variance space. This systematic distortion prevents the accurate recovery of subsequent factors.
In higher-order tensors, this problem compounds. While gradient descent on over-parametrized tensors can sometimes mimic a natural deflation process—as analyzed by Rong Ge et al. in Understanding Deflation Process in Over-parametrized Tensor Decomposition—traditional greedy algorithms break down entirely when the ground-truth components overlap or share statistical support.
Algorithmic Mechanisms for Non-Orthogonal Recovery
If we cannot rely on sequential orthogonal projections, how can we reliably isolate correlated latent factors? Researchers have developed several distinct mechanisms to bypass the 90-degree constraint.
1. Simultaneous Matrix Diagonalization
Rather than treating a tensor as a sequence of isolated vectors, one powerful approach involves reducing the tensor to a set of matrices that can be solved simultaneously. In Tensor Factorization via Matrix Factorization, Volodymyr Kuleshov et al. show that random linear combinations of tensor slices can be transformed into a simultaneous matrix diagonalization problem. Because this approach solves for the entire factor matrix globally rather than greedily vector-by-vector, it successfully uncovers highly correlated, non-orthogonal latent factors that would otherwise confound standard tensor power iterations.
2. Perturbation Bounds and Iterative Projections
Instead of requiring absolute orthogonality, algorithms can operate under relaxed incoherence conditions, which merely bound the maximum allowable correlation between components. Yuefeng Han et al. leverage this in Tensor Principal Component Analysis in High Dimensional CP Models, utilizing iterative projections that remain stable even when data components are highly aligned. By establishing strict perturbation bounds, the authors prove that as long as the components are not completely collinear, the optimization landscape retains enough local curvature to guide the algorithm to the true non-orthogonal factors.
3. Non-Convex Ratio-of-Norms Regularization
In robust matrix and tensor completion, standard convex optimization often relies on the nuclear norm to enforce low-rank structures, which implicitly biases the solution toward orthogonal components. To recover non-orthogonal factors under sparse noise, Huiwen Zheng et al. introduce a non-convex surrogate in Tensor robust principal component analysis via the tensor nuclear over Frobenius norm. By optimizing the ratio of the tensor nuclear norm to the Frobenius norm, the model penalizes redundant dimensions without forcing the remaining basis vectors to be mutually perpendicular, allowing for more flexible data representation.
Soft Geometric Regularization vs. Hard Constraints
Why do these relaxed frameworks perform so much better on real-world data? The answer lies in how they alter the optimization landscape.
When an algorithm enforces a hard orthogonality constraint, it restricts its search space to the Stiefel manifold (the space of orthogonal matrices). If the true physical source signals overlap—such as functional brain networks that share spatial regions (Hanlu Yang 0001 et al.) or sparse matrix decompositions with independent supports (Valeriya Naumova et al.)—the true data-generating solution lies completely outside this manifold. Forcing the model onto it introduces severe structural bias (Haijin Zeng et al.).
Hard Orthogonality Constraint:
[Component 1] ────────────────────────►
│ (Must be exactly 90°)
▼
[Component 2] ────────────────────────►
Relaxed / Incoherent Framework:
[Component 1] ────────────────────────►
\ (Angle θ can vary based on data)
▼
[Component 2] ────────────────────────►
By transitioning to soft geometric regularization—such as regularizing the Euclidean norms of the factor matrices (Jicong Fan et al. in Euclidean-Norm-Induced Schatten-p Quasi-Norm Regularization for Low-Rank Tensor Completion and Tensor Robust Principal Component Analysis) or generalizing robust PCA to overcomplete, non-orthogonal dictionaries (Sirisha Rambhatla et al. in A dictionary based generalization of robust PCA)—the optimization landscape expands.
This flexibility prevents the optimization process from being forced to fit incorrect vectors simply to satisfy rigid geometric constraints. For instance, in subspace recovery, Paris Giampouras et al. demonstrate in Implicit Bias of Projected Subgradient Method Gives Provable Robust Recovery of Subspaces of Unknown Codimension that relaxing mutual orthogonality allows parallel subgradient methods to adaptively find the true codimension of a subspace without being constrained by an overestimation of dimensions.
Theoretical Frontiers
While non-orthogonal factorization algorithms have achieved significant empirical success, providing rigorous theoretical guarantees remains an active area of research.
Many exact recovery proofs for high-dimensional tensor completion still rely on the mathematical convenience of orthogonal components to guarantee that optimization will not get trapped in local minima. As noted by Prateek Jain et al. (Provable Tensor Factorization with Missing Data) and Aaron Potechin et al. (Exact tensor completion with sum-of-squares), extending exact recovery bounds to general, non-orthogonal incoherent structures is a major theoretical challenge.
Similarly, in streaming data contexts like online Independent Component Analysis, standard convergence proofs frequently assume an orthogonal mixing matrix, which necessitates a computationally expensive whitening pre-processing step. Chris Junchi Li et al. (Stochastic Approximation for Online Tensorial Independent Component Analysis) emphasize that eliminating this pre-processing step requires the continued development of stable, non-orthogonal online optimization frameworks.
Key Foundational Work
Block and Group Regularized Sparse Modeling for Dictionary Learning (2013) — contributed 1 concept elements to this direction.
Deflation Methods for Sparse PCA (2008) — contributed 1 concept elements to this direction.