Abstract
We consider the problem of predicting a response variable from a set of covariates on a data set that differs in distribution from the training data. Causal parameters are optimal in terms of predictive accuracy if in the new distribution either many variables are affected by interventions or only some variables are affected, but the perturbations are strong. If the training and test distributions differ by a shift, causal parameters might be too conservative to perform well on the above task. This motivates anchor regression, a method that makes use of exogeneous variables to solve a relaxation of the causal minimax problem by considering a modification of the least-squares loss. The procedure naturally provides an interpolation between the solutions of ordinary least squares and two-stage least squares. We prove that the estimator satisfies predictive guarantees in terms of distributional robustness against shifts in a linear class; these guarantees are valid even if the instrumental variables assumptions are violated. If anchor regression and least squares provide the same answer (anchor stability), we establish that OLS parameters are invariant under certain distributional changes. Anchor regression is shown empirically to improve replicability and protect against distributional shifts.
Introduction
In standard predictive modeling, we often assume that training and test data are independent and identically distributed (i.i.d.). However, real-world data is frequently heterogeneous, collected across varying time periods, locations, or experimental conditions. Ordinary Least Squares (OLS) models tend to overfit to the specific correlations of a training set, leading to poor performance when the environment changes—a phenomenon known as distribution shift.
Causal inference offers a potential solution by identifying underlying mechanisms that should remain invariant under interventions. Yet, pure causal models can be overly conservative, discarding useful non-causal information and resulting in suboptimal predictive performance in stable environments. Anchor regression addresses this by providing a middle ground: a framework that balances predictive power with robustness against distributional perturbations.
Methodology: The Anchor Regression Framework
Anchor regression utilizes exogenous variables, termed "anchors" ($A$), which represent sources of heterogeneity (e.g., batch IDs, locations, or time stamps). These anchors are not the primary targets of prediction but serve to "anchor" the model against shifts.
The core of the method is a modified least-squares objective function controlled by a tuning parameter $\gamma \ge 0$:
- $\gamma = 1$: The model reduces to standard OLS.
- $\gamma \to \infty$: The model converges to Two-Stage Least Squares (2SLS), a fundamental technique in instrumental variable (IV) analysis.
- $1 < \gamma < \infty$: The model provides a balanced solution that interpolates between these two extremes.
Mathematically, anchor regression solves a minimax problem, minimizing the worst-case risk under a class of "shift interventions." By adjusting $\gamma$, practitioners can specify the strength of the perturbations they wish to be robust against.
Causal Interpretations and Flexibility
A key advantage of anchor regression is its ability to handle diverse causal structures. Unlike traditional IV methods that require strict assumptions (e.g., the instrument must not directly affect the outcome), anchor regression provides robustness even when these assumptions are violated.
As shown in the figure above, the anchor variable $A$ can interact with the system in several ways:
- (i) $A$ acts as a standard instrument for the effect of $X$ on $Y$.
- (ii) $A$ directly influences the target $Y$, acting as an "invalid" instrument in the classical sense.
- (iii) $A$ influences a hidden variable $H$ that affects both $X$ and $Y$.
In all these scenarios, anchor regression offers predictive guarantees against shifts in the distribution of $A$.
Empirical Validation
The authors validated anchor regression on two real-world datasets:
- Genotype-Tissue Expression (GTEx): In predicting gene expression across 53 human tissues, anchor regression identified features that were significantly more "stable" and replicable across different tissue types compared to standard Lasso regression.
- Bike Sharing Dataset: Using "date" as an anchor to account for seasonal and daily fluctuations, anchor regression (with $\gamma \approx 2$) improved "worst-case" predictions. It reduced the 90th percentile of prediction error by approximately 10%, making the system more reliable for high-demand, high-variability scenarios.
Conclusion
Anchor regression bridges the gap between OLS and causal inference, offering a tunable framework for distributional robustness. By relaxing the strict requirements of instrumental variables and framing causality as a limit of robustness, it provides a practical tool for building models that generalize better to unseen, heterogeneous environments. Future work includes extending these theoretical guarantees to non-linear models and high-dimensional settings.
Summary updated