Pooling Instrumental-Variable Evidence Across Sites in a Shared Feature Basis

Suppose several hospitals, firms, or data silos each hold confounded observational data on the same causal question, and none may share raw records. Each site also has an instrument: a variable that nudges the treatment but affects the outcome only through it, such as fuel cost shifting ticket prices. Instruments are often strong at some sites and nearly flat at others, so each site can pin down the causal response curve only over part of its shape. This proposal shows how to combine their evidence anyway. Each site trains a deep instrumental-variable model locally. The sites then agree on a common coordinate system for their learned treatment features. To build it, they place orthonormalized feature evaluations on a shared, outcome-free anchor set and average the orthogonal projectors across sites. Finally, they add up the tiny second-stage regression statistics in that common basis. The result is a one-shot, communication-light estimator whose pooled second stage is well conditioned even when every individual site's second stage is not.

The core claim fits in two sentences. In federated causal estimation with deep features, neither averaging network weights nor averaging final predictions is the right operation. The right operation is to sum second-stage moment statistics, and that only makes sense after the sites' feature coordinates are synchronized, which the orthogonal-alignment literature already knows how to do. We expect the gain to be largest where existing methods are weakest: when instrument strength varies across sites, so that identification exists only in the union of the sites.

Background and the gap

Two-stage instrumental-variable (IV) estimation works in two steps. The first stage learns how the instrument $Z$ moves the treatment $X$. The second stage regresses the outcome $Y$ on that instrument-predicted part of the treatment, which removes the influence of unobserved confounders.

This proposal builds on a specific line of IV methods. Kernel Instrumental Variable Regression does both stages with kernel ridge regression and is minimax optimal, but it scales cubically in sample size. Deep-feature IV replaces the fixed kernel with learned features: a treatment network $\psi(x)$, an instrument network $\phi(z)$, a linear stage-1 map from $\phi$ to $\psi$, and a linear stage-2 head. Optimality and Adaptivity of Deep Neural Features for Instrumental Variable Regression proves these learned features adapt to spatially uneven response curves where any fixed kernel or sieve cannot. That result supplies our method and also motivates keeping features learned rather than fixed. Deep Proxy Causal Learning uses the same two-stage feature structure for proxy variables and was the only method that worked reliably on 4,096-dimensional dSprite images. Regularized DeepIV with Model Selection shows that ridge-regularized two-stage regression avoids the instability of min-max formulations and supports principled model selection. That result constrains our design: we deliberately avoid a federated adversarial game.

The only federated IV work in the file is Federated Instrumental Variable Analysis via Federated Generalized Method of Moments. It runs DeepGMM, a min-max formulation, as a federated game with weight averaging, and proves its error grows with client gradient dissimilarity. It never addresses the fact that locally learned representations live in different coordinate frames.

The alignment literature supplies the missing operation. Communication-efficient distributed eigenspace estimation aligns local bases to a reference by orthogonal Procrustes (finding the rotation that best matches one basis to another) before averaging. It proves that the error added by alignment is quadratic in local estimation error. Distributed estimation of principal eigenspaces averages projectors $UU^\top$, which do not depend on the choice of basis, and shows this one-shot estimator reaches centralized rates once each site has more samples than a critical mass. FedRot-LoRA shows that rotational drift corrupts federated averaging of low-rank fine-tuning factors and that Procrustes alignment fixes it.

All three papers align bases for unsupervised or fine-tuning objectives. None of them asks what happens when the aligned objects feed a downstream regression that is ill-posed, meaning small errors in the inputs can be strongly amplified, as the IV second stage is.

The closest paper by name, Orthogonal Representation Learning for Estimating Causal Quantities, uses "orthogonal" in the statistical, Neyman sense: the loss is insensitive to errors in auxiliary nuisance estimates. It does not use the geometric sense of rotations. It is also centralized and assumes no unobserved confounding, so it does not overlap with this proposal beyond motivating representation-aware causal estimation.

Core Claim and Mechanism

Site $i$ holds $n_i$ samples and trains deep-feature IV locally, producing treatment features $\psi_i:\mathcal{X}\to\mathbb{R}^K$. Learned features are identified only up to an invertible linear map, so $\psi_i$ and $\psi_j$ are not comparable coordinate by coordinate.

All sites evaluate their features on a shared anchor set $A$ of $n_a$ treatment values. These anchors carry no outcomes or instruments: they can be a grid, public unlabeled images, or server-generated points. Each site orthonormalizes its evaluations into $U_i\in\mathbb{R}^{n_a\times K}$. After this whitening, two sites whose features span the same function space differ only by a rotation.

The server forms $\bar P=\frac{1}{m}\sum_i U_iU_i^\top$ and takes its top-$K$ eigenvectors $U^\star$, in the manner of Fan et al. Each site then solves a Procrustes problem from $U_i$ to $U^\star$, which gives a fixed linear map $T_i$ placing $\psi_i$ in consensus coordinates.

Stage 1 stays local. Each site regresses its aligned treatment features on its own instrument features and obtains predicted features $\hat\psi_i(z)=\hat W_i\phi_i(z)$ in the consensus basis. Instrument features therefore never need aligning. The pooled second stage is

$$ \hat\beta=\Big(\sum_{i=1}^m \hat\Psi_i^\top\hat\Psi_i+\lambda I_K\Big)^{-1}\sum_{i=1}^m \hat\Psi_i^\top Y_i, $$

where $\hat\Psi_i$ stacks $\hat\psi_i(z)$ over site $i$'s samples, $Y_i$ stacks its outcomes, and $\lambda$ is the ridge penalty. The estimated structural function at a site is $\hat h(x)=\hat\beta^\top T_i\psi_i(x)$. Each site sends a $K\times K$ matrix, a $K$-vector, and $n_a\times K$ anchor evaluations.

The mechanism matters because of heterogeneous instrument strength. A site with a weak or range-restricted instrument has a nearly singular local Gram matrix $\hat\Psi_i^\top\hat\Psi_i$. Its own ridge solution shrinks $h$ toward zero in the poorly identified directions. Averaging local predictions therefore inherits that shrinkage in proportion to the share of weak sites. Summing Gram matrices instead weights each direction by how much information each site actually has about it, which mirrors how centralized two-stage least squares would combine the data.

We state as a hypothesis to be checked, not a proven result, that the alignment-induced excess risk scales like the squared local subspace error divided by the smallest eigenvalue of the pooled Gram matrix. The squared numerator follows Charisopoulos et al.'s quadratic benefit. The denominator is the usual ill-posedness amplification in IV.

Projector averaging is chosen over Procrustes-to-one-reference for a specific reason. When the designated reference site has a weak instrument, its feature space is the worst one to align to. The file identifies this as an open failure mode of reference-based alignment.

Research Program

The work has three parts. The first is building the estimator. This includes a variant that weights each site's projector by its stage-1 fit, and a variant that also pools stage 1 using instrument-space anchors. The second is a heterogeneity simulator that varies three things independently: the fraction of weak-instrument sites, how much each site's instrument range overlaps with the others, and Dirichlet label skew as in the FEDIV benchmarks. The third is an analysis that measures, for each site, the principal angles between its feature span and the consensus span. This tells us when a site's features cannot be reconciled by rotation, a case where an orthogonal map is the wrong tool and a general linear map or exclusion is needed.

We will also measure the critical local sample size below which alignment fails. Fan et al. predict such a threshold, and we will test whether instrument weakness pushes it upward.

Baselines

The naive reference points are local-only estimation, reporting both the best single site and the average site, and the centralized deep-feature IV oracle trained on pooled data. These bracket the attainable range. Success means closing most of the gap between them.

Prediction averaging trains a model at each site and averages the final $\hat h_i$. It is immune to rotation by construction. Matching or beating it isolates the benefit of pooling statistics rather than outputs.

Federated averaging of the deep-feature IV networks, and FEDDEEPGMM from Federated Instrumental Variable Analysis via Federated Generalized Method of Moments, are the strongest existing federated methods. On their 1D linear design with Dirichlet $\alpha=0.3$, that paper reports a test MSE of 0.04 for federated DeepGMM against 0.03 for centralized DeepGMM. On the Demand, MNIST-demand, and dSprite designs, no reported result was found on Lacuna for federated methods. We will run FEDDEEPGMM ourselves with its published settings, the same client partitions, and the same total gradient-step budget.

Shared random-feature kernel IV gives every site the same random Fourier features from a common seed and sums exact sufficient statistics. It needs no alignment at all.

Two ablations isolate our key ingredient. The first pools the second-stage statistics without alignment. The second aligns by Procrustes to a single reference site, in the style of FedRot-LoRA, instead of averaging projectors.

All methods get the same per-site data, the same anchor set where relevant, and equal hyperparameter search budgets. Selection uses held-out stage-2 loss at each site, following the model-selection logic of Li et al. Beating the strongest competitor means lower structural-function MSE than both FEDDEEPGMM and prediction averaging across the weak-site sweep, with non-overlapping confidence intervals over 20 seeds.

Why not just…?

Why not just give every site the same fixed random features and sum sufficient statistics exactly? This is the most serious threat. On low-dimensional designs it may tie or beat us, because the pooling is exact and no alignment error exists. Its weakness is that fixed features cannot adapt to uneven response curves, as Kim et al. prove. They also failed on image covariates, where Xu et al. report that "traditional kernel methods were overwhelmed." Our experiments compare the two directly across input dimension. If shared random features match us on MNIST-demand and dSprite, we will report that the learned-feature-plus-alignment route is unnecessary.

Why not just average the sites' predictions? Averaging outputs avoids rotations entirely. However, each weak site's ridge solution is biased toward zero in the directions it cannot identify, and averaging preserves that bias. The weak-site-fraction sweep is designed to show whether this bias grows as predicted. If it does not, the pooling argument fails.

Why not just run federated averaging or FEDIV on the networks? The FEDIV paper's own theorem ties its error to client dissimilarity. Federated averaging ignores the feature-coordinate drift that FedRot-LoRA showed costs about 10 to 12 points on HumanEval and MNLI in the low-rank fine-tuning setting. That paper reports 28.77% against 40.88% pass@1 on HumanEval and 76.80% against 87.34% accuracy on MNLI. We test whether the analogous loss appears in causal estimation.

Why not align to one reference site? It is cheaper, and it may suffice when sites are homogeneous. The reference-site ablation, run with the reference deliberately chosen to be a weak-instrument site, tests whether projector averaging is actually needed.

Proposed Experiments

We will use the Demand design from Hartford et al. and Singh et al., split across $m\in\{2,4,8,16,32\}$ sites. The fuel-cost instrument at each site is restricted to a sub-interval, and we vary both how much these intervals overlap and the confounding strength $\rho$. We will use the FEDIV 1D Linear, Absolute, and Step designs to make direct contact with the 0.04 number above. For high-dimensional inputs we will use MNIST-demand, and we will use the dSprite proxy benchmark to test whether the same pooling carries over to proxy causal learning. The metric throughout is out-of-sample MSE of the structural function on a held-out treatment grid.

Four checks are decisive. The first: pooling without alignment should collapse toward the local-only error, while aligned pooling should track the centralized oracle. The second: as the fraction of weak sites rises, prediction averaging should degrade and our estimator should not. The third: varying anchor count and shifting the anchors away from the treatment distribution should locate where alignment breaks. The fourth: shrinking local sample size should reveal a critical-mass threshold, and we will check whether it rises as instruments weaken. If the second check fails, the core claim is wrong.