Multiply Robust Causal Adjustment with Compressed Video Features

When a platform, a clinic, or a driving system decides who gets a treatment partly based on what a video shows, the video becomes a confounder: it influences both who is treated and how things turn out. The standard move is to run the video through a large pre-trained encoder, compress the resulting embedding to a handful of dimensions, and adjust for those. The last step quietly assumes that the compressed features still contain every piece of the video that drives treatment and outcome. Nobody checks this. A small background cue that shaped the treatment decision can sit in a low-variance direction that compression throws away, and the resulting effect estimate is biased with nothing to flag it.

This note proposes to treat compression as part of the causal estimator. We fit each nuisance model on its own supervised low-rank projection of the video embedding. A nuisance model is one of the auxiliary regressions that an effect estimator needs, such as the probability of treatment given the video. We combine these models in a multiply robust estimator, which stays consistent as long as at least one model in each of several pairs is correct. The product structure of its bias means a projection only has to keep what one partner in each pair needs, which is a sharper and more checkable requirement than "keep everything." Refitting the estimator under different ways of assigning projections to models then yields a diagnostic: if the assignments disagree, some projection has dropped a confounder.

Core Claim

Doubly robust estimators have bias proportional to the product of the error in the treatment model and the error in the outcome model. Suppose the video embedding $X \in \mathbb{R}^d$ is compressed differently for each model: the outcome regression sees $A^\top X$ and the propensity model (the model of treatment probability) sees $B^\top X$, where $A, B$ are $d \times k$ projection matrices with $k \ll d$. The leading bias of the average-effect estimate is then

$$ \text{Bias} \approx \mathbb{E}\left[\frac{\big(e(X) - \hat e(B^\top X)\big)\big(\mu(X) - \hat\mu(A^\top X)\big)}{\hat e(B^\top X)}\right], $$

where $e(X)$ is the true probability of treatment given the full embedding, $\mu(X)$ is the true outcome regression, and hats denote fitted models. Two consequences follow. First, the bias vanishes if $A$ captures everything the outcome depends on, or if $B$ captures everything the treatment depends on. It does not require a single projection that captures both. Second, when the estimator has more nuisance models, the bias becomes a sum of such products. This happens, for example, in the quadruply robust estimator of Policy Learning for Balancing Short-Term and Long-Term Rewards, which combines propensity, selection-into-observation, and two outcome regressions when long-term outcomes are missing. In that case the question "how many dimensions do I keep?" becomes a question of allocating rank across models so that every product has at least one well-covered factor.

The claim we will test has three parts. Supervised, model-specific projections give lower bias and better confidence-interval coverage than a single shared unsupervised projection at the same total rank. The estimator's agreement across projection assignments is a usable test for compression-induced confounding. And rotating the embedding into these learned bases is what makes tree ensembles, which split on one coordinate at a time, workable on video features at all.

Research Program

The pipeline has four stages. Each is simple on its own; the contribution is the bias analysis and the diagnostic that ties them together.

First, we extract clip embeddings from pre-trained video encoders. We use the Two-Stream Inflated 3D ConvNet of Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset, which inflates 2D image filters into 3D filters over time and is pre-trained on Kinetics. We also use 3D residual networks from Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?. Both play the role of fixed feature suppliers. The file documents that such features are dense and noisy, and prone to background shortcuts (CausalVTG, VACT). Shortcuts are harmful for recognition but exactly what we must adjust for when they drive treatment.

Second, we learn a separate low-rank projection for each nuisance target. For the outcome we use reduced-rank regression of $Y$ on $X$, and for the treatment the analogous reduced-rank logistic fit of $T$ on $X$. For the selection model we fit the missingness indicator. This departs from the unsupervised compressions surveyed in the file. Representing Sentences as Low-Rank Subspaces shows rank 3–5 subspaces capture 70–90% of the variance of token embeddings. Get What You Want, Not What You Don't uses a singular value decomposition to isolate and suppress semantic directions. Both motivate low-rank structure. But variance captured is the wrong criterion here, because confounding can live in low-variance directions. We keep their geometric tool, principal angles between subspaces, as an analysis instrument: the angle between the learned $A$ and $B$ tells us how much the treatment-relevant and outcome-relevant parts of the video overlap.

Third, we fit each nuisance model with honest random forests on its projected coordinates. We also estimate heterogeneous effects with the orthogonal forest of Orthogonal Random Forest for Causal Inference. That paper combines score functions that are insensitive to small nuisance errors with forest splitting, and proves oracle rates under local sparsity. Its authors flag that local sparsity is the assumption dense embeddings break. We restore it by construction, since in a learned $k$-dimensional basis the relevant signal is concentrated in few coordinates. We also replace Forest Lasso with plain forests, because $k$ is small. Out-of-bag predictions stand in for cross-fitting, which addresses the computational cost the file notes for nested estimators with bootstraps.

Fourth, we build the multiply robust estimator from the relevant efficient influence function, which is the per-example correction term that gives the smallest possible asymptotic variance. We use the quadruply robust form for the missing-long-term-outcome setting. We then compute the assignment diagnostic: the same estimator is refit with projections swapped or shared across models, and a multiplier bootstrap tests whether the estimates differ. The bootstrap follows Multiply-Robust Causal Change Attribution, which pairs multiply robust estimation with a multiplier bootstrap for inference. On the theory side we will state conditions on the projection error, meaning the gap between $\mu(X)$ and its best approximation from $A^\top X$, under which the product bias is $o(n^{-1/2})$. This rate is what matters because the estimator's statistical noise shrinks as $n^{-1/2}$, so a smaller bias does not distort the confidence intervals. We also state conditions under which the diagnostic has power against a dropped confounding direction.

Baselines

The naive reference point adjusts for nothing video-specific. It uses only the tabular covariates and treats the video content as ignorable. The gap between this and any video-aware estimator measures how much confounding the video carries. In semi-synthetic settings where we plant that confounding, its bias is known.

The strongest existing competitor is cross-fitted double machine learning, with gradient-boosted trees as nuisance learners, fitted on the full raw embedding. This is how a careful applied researcher would proceed today, and it involves no compression, so it cannot lose a confounder by compressing. Multiply-Robust Causal Change Attribution uses exactly this style of black-box gradient-boosting nuisance fits. We also run its multiply robust extension on raw embeddings, the quadruply robust estimator of Policy Learning for Balancing Short-Term and Long-Term Rewards with boosted nuisances.

For heterogeneous effects, the strongest competitor is the orthogonal random forest applied to principal-component features at the same total rank. For published reference numbers, the only scalar reported for the quadruply robust policy is on JOBS: a balanced reward of about 2895, versus 1698 for a short-term-only and 1260 for a long-term-only policy. That is a tabular dataset. For its KuaiRec experiment, no scalar value is reported. No reported result was found on Lacuna for any causal estimator using video-content features on KuaiRec, Something-Something, or Kinetics. The orthogonal forest paper reports stable RMSE relative to DML and GRF in its 500-control simulation but no scalar values in the file. So our baselines must be rerun rather than quoted.

For fairness, all methods see the same embeddings, sample splits, and tabular covariates. Each gets the same hyperparameter search budget, counted in nuisance-model fits. Total rank is matched when comparing compressed variants.

Three ablations isolate the key ingredient. The first uses one shared supervised projection for all models, which tests whether separate projections matter. The second uses one shared unsupervised principal-component projection, which tests whether supervision matters. The third uses a doubly robust instead of quadruply robust estimator on the same projections, which tests whether the extra nuisance pairs matter. We would count the method as beating the baselines if it matches raw-embedding double machine learning on RMSE while achieving nominal 95% interval coverage where that baseline under-covers, or if it beats every compressed variant on bias at equal rank. A win would also count only if the assignment diagnostic flags planted dropped confounders with controlled false-alarm rate.

Why not just…?

Why not just run double machine learning on the full embedding and skip compression? This is the real threat. With large samples, boosted trees on 512–2048 dimensions may estimate nuisances well enough that compression adds nothing. The file's evidence cuts both ways. Concept-driven Off Policy Evaluation reports variance reductions of one to two orders of magnitude from mapping continuous states into a few discrete concepts, and the orthogonal forest paper reports RMSE spikes for DML as confounder support grows. But neither involves video embeddings. We settle it by sweeping sample size on the semi-synthetic benchmark. If raw-embedding DML matches our estimator in both RMSE and coverage at every sample size we test, the estimation-accuracy claim fails. The diagnostic would then be the remaining contribution, and we would report that honestly.

Why not just keep more principal components until the estimate stops changing? A rank sweep is cheap and is the natural practitioner check. It fails in a specific way: it adds directions in order of variance. A confounding cue in a low-variance direction therefore enters late or not at all, and the estimate can look stable across many ranks while still biased. Our semi-synthetic design plants confounding in controlled variance quantiles of the embedding precisely to test this. If rank sweeps catch the planted confounders as reliably as the assignment diagnostic, we drop the diagnostic claim.

Why not just train an end-to-end neural effect estimator on the frames? A network with separate heads for treated and untreated outcomes could learn its own representation. Image-based Treatment Effect Heterogeneity found such a post-hoc neural approach inferior to structured clustering at recovering effect clusters from satellite images, which argues against it. That was a static-image setting, though. More importantly, an end-to-end network offers no product-form bias guarantee and no confidence intervals. We include a fine-tuned neural head on the same embeddings as an extra comparison, judged on coverage as well as RMSE.

Proposed Experiments

The primary testbed is semi-synthetic. We take real clips from Something-Something, whose templated physical actions give a meaningful "what happens" subspace, and from Kinetics. We simulate treatment from one set of embedding directions, outcome from a partly overlapping set, and missingness of a long-term outcome from a third. We vary the overlap angle and the variance quantile of the confounding directions. Ground-truth effects are then known, so we report bias, RMSE, and 95% coverage for the average effect. For heterogeneous effects we report the error between estimated and true per-clip effects. We report the diagnostic's detection rate and false-alarm rate against planted omissions.

The decisive checks are three. The first compares separate-projection and shared-projection estimators at equal total rank when the treatment and outcome subspaces are nearly orthogonal. This is where the product bias argument predicts the largest gap. The second checks whether the diagnostic fires when, and only when, a planted confounding direction is outside both relevant projections. The third checks whether forests on learned coordinates beat forests on principal components for heterogeneous effects, which tests the axis-alignment argument.

The real-data study uses KuaiRec (KuaiRec), whose near-fully-observed small matrix (99.6% density) lets us mask long-term outcomes and still score policies against observed truth. This reproduces the missing-ratio sweep from 0.1 to 0.6 of the quadruply robust paper, with video content added as a confounder. One caveat: the file does not confirm that raw video frames are distributed with KuaiRec. If they are not, we will embed the available item content (captions and tags) and run the identical analysis. The mechanism is the same, but the video-specific conclusions then rest on the semi-synthetic benchmark alone.