Abstract
Tensor regression is a powerful tool for high-dimensional data analysis, yet it is frequently undermined by "CP degeneracy"—a phenomenon where the best low-rank approximation of a tensor does not exist. This paper characterizes the impact of degeneracy on statistical regression and proposes a penalized M-estimation framework. By penalizing the individual factors of the CANDECOMP/PARAFAC (CP) decomposition rather than the reconstructed tensor, we provide a theoretical guarantee for the existence of a stable estimator and establish its convergence rates.
Introduction
In modern data science, multi-dimensional arrays, or tensors, are ubiquitous. To mitigate the "curse of dimensionality," researchers often assume that the underlying coefficient tensor in a regression model is low-rank. The CANDECOMP/PARAFAC (CP) decomposition is the most common approach, representing a high-dimensional tensor as a sum of $R$ rank-1 components.
However, tensor rank possesses topological properties that differ significantly from matrix rank. Most notably, the set of tensors with rank at most $R$ is not closed. This leads to the problem of CP degeneracy: a sequence of rank-$R$ tensors can converge to a tensor of rank $R+1$. When a regression target falls within this gap, optimization algorithms attempt to "chase" an unattainable limit, causing model parameters to diverge to infinity and numerical solvers to fail.
The Problem of CP Degeneracy
While CP degeneracy was previously recognized in the context of tensor approximation, its implications for statistical regression remained poorly understood. The authors demonstrate that if a solution to a tensor regression problem is unattainable, the magnitude of the CP parameters must diverge. This divergence is typically accompanied by component matrices becoming nearly rank-deficient, creating numerical instability that renders standard estimators useless.
Proposed Methodology: Penalized M-Estimation
To address this instability, the paper introduces a generalized penalized framework designed to ensure the existence of a solution even in the presence of potential degeneracy. The proposed estimator minimizes the following objective:
$$\min_{\theta} f(\theta) + \lambda g(\theta)$$
Where $f(\theta)$ is the loss function and $g(\theta)$ is a penalty applied directly to the CP parameters (the individual vectors comprising the decomposition).
The Parameterization Insight
A critical finding of this research is that the choice of penalty target is paramount. Standard regularization techniques, such as a ridge penalty on the reconstructed tensor entries, fail to prevent degeneracy. To maintain optimization within a compact space where a solution is guaranteed to exist, one must "reach inside" the decomposition and penalize the factors ($\theta$) themselves.
Theoretical Guarantees
The paper provides two primary theoretical contributions:
- Existence Guarantee: The authors prove that if the penalty function $g(\theta)$ is continuous and diverges as the parameters increase, the optimization problem is guaranteed to have a stable solution (Corollary 1).
- Convergence Rates: The authors establish an upper bound for the estimation error: $$\|\hat{\mathbf{A}} - \mathbf{A}_0\|_F^2 \leq O_p\left(\frac{R^D + R \sum p_d}{n} + \delta^2 + \frac{\lambda G}{n}\right)$$ This bound demonstrates that the error scales predictably with the tensor dimensions ($p_d$), the rank ($R$), and the sample size ($n$), providing a safety net for practitioners working with high-dimensional data.
Numerical Results
The framework was validated using synthetic datasets designed to trigger degeneracy in both noiseless and noisy environments. The experiments compared standard Least Squares, coefficient-tensor penalization, and the proposed CP-parameter penalization.
In simulations where the "true" underlying tensor was degenerate, standard Least Squares and tensor-penalized methods diverged in 100% of trials. Conversely, the CP-parameter penalization method converged to a stable, well-defined solution in every instance, demonstrating its robustness against structural mismatches between the model and the data.
Conclusion and Future Work
This work establishes a formal link between the geometric non-closedness of tensor spaces and the failure of regression algorithms. By shifting the focus of regularization from the reconstructed tensor to its constituent factors, the authors provide a mathematically sound method for avoiding exploding parameters.
Future research may explore whether other tensor formats, such as Tucker or Tensor Train (TT), exhibit similar degenerate behaviors. Additionally, while the current study utilizes synthetic data, applying CP-parameter penalization to real-world multi-way data, such as neuroimaging or longitudinal studies, remains a vital next step for validating these theoretical gains in practical settings.
Summary updated