Abstract: We study how correlations in the design matrix influence Lasso prediction. First, we argue that the higher the correlations, the smaller the optimal tuning parameter. This implies in particular that the standard tuning parameters, that do not depend on the design matrix, are not favorable. Furthermore, we argue that Lasso prediction works well for any degree of correlations if suitable tuning parameters are chosen. We study these two subjects theoretically as well as with simulations.
Introduction
The Lasso (Least Absolute Shrinkage and Selection Operator) is a cornerstone of high-dimensional statistics, particularly when the number of variables ($p$) exceeds the number of observations ($n$). While the Lasso is widely used, its theoretical guarantees often rely on "fast rate" bounds that require the design matrix to have weakly correlated variables. In practice, however, real-world data is frequently characterized by high degrees of redundancy. This paper investigates whether such correlations inherently degrade Lasso performance or if the issue lies in how we tune the model.
Redefining Correlation in High Dimensions
To analyze the impact of design complexity, we introduce two measures of correlation, $K_\kappa$ and $F$, derived from metric entropy and covering numbers. These factors quantify the "effective" dimensionality of the design. If many columns in a design matrix can be closely approximated by a small subset, the search space for the Lasso is effectively reduced. This theoretical innovation allows us to move beyond the pessimistic "slow rate" bounds typically associated with correlated designs.
The Correlation-Tuning Trade-off
A central finding of this research is that the optimal tuning parameter $\lambda$ is not a static value determined solely by $n$ and $p$. Instead, it is deeply dependent on the design's correlation structure. Specifically, as correlation increases, the optimal $\lambda$ decreases.
Standard practice often sets $\lambda \propto \sqrt{\log p / n}$, but our simulations demonstrate that this choice is sub-optimal when variables are redundant. High correlation effectively shields the model from noise by reducing the complexity of the stochastic term. Consequently, a lighter penalty (smaller $\lambda$) is sufficient to prevent overfitting while maintaining predictive accuracy.
As shown in the figures above, when the correlation between variables is extremely high (Figure 1), the optimal tuning parameter remains low. As the "impertinent" variables become noisier and thus less correlated with the original design (Figure 2), the optimal $\lambda$ for the expanded design shifts significantly to the right.
Prediction vs. Selection
It is crucial to distinguish between prediction error ($X\beta$) and variable selection (identifying the true $\beta_0$). While high correlation makes it difficult to identify exactly which features are responsible for a signal, it does not necessarily harm the ability to predict the response. If two variables are nearly identical, the Lasso's choice between them has negligible impact on the final prediction. Our results show that despite adding massive amounts of redundant variables, the minimal prediction error remains remarkably stable, provided $\lambda$ is adjusted downward.
Conclusion
Correlations are not an inherent enemy of the Lasso. Rather, they necessitate a more nuanced approach to tuning. Because the optimal $\lambda$ drops as correlation increases, practitioners should rely on design-dependent tuning methods. While we do not provide a closed-form formula for this adjustment, cross-validation remains the most robust practical tool for identifying the optimal, design-sensitive $\lambda$ in real-world applications.
Summary updated