Imagine trying to determine the orientation of a flat tabletop based on a set of data points. If your data points are scattered across the entire surface, the orientation is easy to pin down. But if all your data points happen to lie along a single straight line, you can tilt the table in infinitely many directions around that line, and it will still fit the data perfectly.
This is the geometric intuition behind multicollinearity—a condition where explanatory variables in a statistical model are highly correlated with one another. In high-dimensional settings where the number of features significantly exceeds the number of samples, this problem becomes acute. Highly correlated features create a mathematical "nullspace" where many different combinations of feature weights can produce the exact same predictive output. Consequently, models lose their ability to isolate the individual impact of specific drivers, leading to severe instability, high variance, and uninterpretable models.
The Illusion of Identifiability
When features are nearly perfectly correlated, statistical frameworks struggle fundamentally with identifiability. As noted by Shaer et al. in their work on conditional randomization tests (Learning to Increase the Power of Conditional Randomization Tests), collinear features look mathematically identical to a model. This creates an inherent limitation: even if a predictive model achieves exceptional accuracy, it remains impossible to definitively isolate which specific variable is the true causal driver.
This lack of identifiability directly undermines the interpretation of model coefficients. In an ideal linear model, a coefficient represents the change in the response variable for a one-unit change in a predictor, holding all other variables constant. But when variables are tightly coupled, you cannot mathematically hold one constant while changing the other. As a result, standard regression models encounter extreme variance, where minor fluctuations in the training data cause coefficient estimates to swing wildly or even flip signs, as observed by Ryu et al. (Evaluating Variable Selection Techniques for Multivariate Linear Regression) and Pitt (Application of bivariate negative binomial regression model in analysing insurance count data).
Why Standard Regularization Fails
A common remedy for high-dimensional data is to apply sparsity-inducing regularization like the LASSO. However, multicollinearity breaks the core mechanics of these selectors.
As explained by Li et al. (Graph-Based Regularization for Regression Problems with Highly-Correlated Designs), if two variables behave almost identically, LASSO will arbitrarily select one and discard the other. This arbitrary behavior means that if you resample your data slightly, the model might select an entirely different, scattered subset of features. Obuchi et al. demonstrate that this redundancy prevents models from distinguishing between true signals and collinear noise, leading to highly unstable variable selection (Semi-Analytic Resampling in Lasso). Similarly, Lee et al. show that traditional Bayesian shrinkage techniques suffer from dramatic increases in error under multicollinearity because they struggle to allocate probability mass among competing, correlated predictors (Tail-adaptive Bayesian shrinkage).
Without Collinearity:
Feature 1: ───[Signal]───► Distinct Information
Feature 2: ───[Signal]───► Distinct Information
(Stable, identifiable coefficients)
With Collinearity:
Feature 1: ───[Signal]───┐
├─► Shared Variance (Nullspace)
Feature 2: ───[Signal]───┘
(Infinite weight combinations; LASSO picks one arbitrarily)
Exploding Weights and Black Boxes
The mathematical consequence of an ill-conditioned design matrix is that the model's optimization landscape becomes a flat valley rather than a sharp peak. In ensemble learning, this redundancy causes serious issues. Merz et al. show that when individual neural networks in an ensemble produce highly correlated outputs, standard techniques for calculating optimal weights "explode" (Combining Neural Network Regression Estimates with Regularized Linear Weights). Because the models are redundant, the optimizer attempts to cancel out shared errors by assigning massive positive and negative weights to different models, making the ensemble highly sensitive to noise.
In high-dimensional linear regression, this phenomenon effectively turns transparent models into black boxes. Schaeffer et al. demonstrate that under extreme multicollinearity, regression coefficients cease to reflect the true underlying features and instead reflect global data variance or regularization artifacts (Interpretation of High-Dimensional Regression Coefficients by Comparison with Linearized Compressing Features).
This instability manifests across diverse domains:
- Neuroimaging: Sun et al. show that extreme feature correlation causes multivariate models to select scattered, unstable feature subsets, degrading the clinical utility of diagnostic brain maps (FDR-HS: An Empirical Bayesian Identification of Heterogenous Features in Neuroimage Analysis).
- Causal Inference: Cousineau et al. find that optimization-based balancing methods frequently suffer from computational failures or produce highly unstable weights when trying to balance treatment groups with correlated observational data (Estimating causal effects with optimization-based methods: A review and empirical comparison).
- Physical Systems: In power grid modeling, Liu et al. observe that highly correlated voltage measurements at adjacent nodes make standard regression approaches mathematically unstable and ill-conditioned (Data-Driven Power Flow Linearization: A Regression Approach).
Restoring Stability Through Dimensionality Reduction
To resolve the ambiguity of the nullspace, researchers must alter the geometry of the problem. The most effective way to do this is to project the highly correlated, high-dimensional feature space onto a lower-dimensional space composed of orthogonal (uncorrelated) components.
Correlated Features (X1, X2) ──► Projection (SVD/PCR) ──► Orthogonal Components (Z1, Z2)
│
▼
Stable Regression Weights
Principal Component Regression (PCR) is a foundational tool for this task. By extracting the principal components of the design matrix, PCR discards the directions of negligible variance where noise overfits, isolating the most significant underlying signals. Bhaskara et al. utilize this property to maintain predictive reliability even when data is missing or adversarially corrupted (Principal Component Regression with Semirandom Observations via Matrix Completion), while Han et al. extend this framework to handle heavy-tailed, non-Gaussian distributions (Robust Sparse Principal Component Regression under the High Dimensional Elliptical Model).
As a computationally efficient alternative to the Singular Value Decomposition required by PCR, Slawski demonstrates that Compressed Least Squares (CLS) regression can stabilize models by compressing the feature space, preventing the model from overfitting to collinear noise (Compressed Least Squares Regression revisited).
Rather than projecting onto abstract mathematical axes, another strategy is to group correlated features based on domain knowledge or statistical clustering. Harris et al. implement this by clustering highly correlated genes into "meta-covariates" (Inferring Meta-covariates in Classification). By running probit regression on these meta-covariates instead of individual genes, the authors eliminate multicollinearity, achieving stable parameter estimation while preserving biological interpretability.
Key Foundational Work
Inferring Meta-covariates in Classification (2009) — contributed 1 concept elements to this direction.
Combining Neural Network Regression Estimates with Regularized Linear Weights (1996) — contributed 1 concept elements to this direction.