Causal Transport Alignment for Streaming Video Domain Adaptation

A video model that must commit to a prediction while the video is still playing, such as an early action detector, a driver-monitoring system or a robot watching a human, only ever sees a prefix of the clip. When such a model moves to a new domain (new cameras, new people, new kitchens), today's adaptation methods align the source and target domains by comparing whole clips or bags of frames. That comparison ignores when information shows up. Two domains can look nearly identical as collections of complete clips while differing sharply in how early the telling motion appears. A predictor that acts on prefixes is exactly the kind of system that breaks under this second kind of shift. Mathematical finance has a tool for this problem: the adapted Wasserstein distance, a version of optimal transport (the cheapest way to move one probability distribution onto another) in which the matching at time $t$ may use only what has happened up to time $t$. It is known to be the right notion of closeness whenever decisions are made sequentially.

We propose to use the adapted Wasserstein distance, computed on a small number of temporally compressed "key steps," as the alignment loss for unsupervised domain adaptation of streaming video models. Temporal redundancy is what makes this affordable. Most frames add almost no new information, so the exact causal transport problem only has to be solved at the few steps where information changes. The adaptation game is also more stable than a gradient-reversal discriminator, because the "adversary" is an optimal coupling computed by a convergent solver rather than a second network trained in a minimax loop. The central prediction can be falsified: gains over ordinary (non-causal) Wasserstein alignment should be largest at early observation ratios and should shrink to nothing once the full clip is seen.

Core Claim

For a predictor that outputs a decision at every step from the prefix it has seen, the relevant measure of domain gap is not the Wasserstein distance between clip-level feature distributions. It is the distance between the two feature processes under causal couplings. The adapted-topology literature shows that ordinary Wasserstein distance can stay small while sequential decision quantities such as optimal stopping values change discontinuously. Approximation of martingale couplings on the line in the adapted weak topology establishes the adapted weak topology as the setting in which such quantities are stable. A note on the adapted weak topology in discrete time shows that the competing filtration-aware topologies coincide and are metrized by the adapted Wasserstein distance. We plan to carry this into domain adaptation as a bound. We conjecture, and will try to prove, that the target risk of a Lipschitz prefix-predictor is at most its source risk plus a constant times the adapted distance between source and target feature processes, plus a joint-error term. We then plan to minimize that distance directly.

The claim is narrow on purpose. For offline whole-clip recognition, a non-causal coupling is not "cheating," and we do not expect causal alignment to help there. The proposal lives or dies on the streaming and early-recognition regime.

Research Program

The adapted distance has a dynamic programming structure. Causal transport in discrete time and applications (Theorem 2.7) shows that bicausal transport between processes $\mu$ and $\nu$ on $T$ steps decomposes backward in time:

$$V_t(x_{\le t}, y_{\le t}) = \inf_{\pi_t \in \Pi(\mu_{t+1}(\cdot \mid x_{\le t}),\, \nu_{t+1}(\cdot \mid y_{\le t}))} \mathbb{E}_{\pi_t}\big[c(x_{t+1}, y_{t+1}) + V_{t+1}(x_{\le t+1}, y_{\le t+1})\big].$$

Here $x_{\le t}$ and $y_{\le t}$ are source and target feature histories. $\mu_{t+1}(\cdot \mid x_{\le t})$ is the source distribution of the next feature given the history, and $\nu_{t+1}(\cdot \mid y_{\le t})$ is its target counterpart. $c$ is the per-step cost, and $V_0$ is the adapted distance. Each step matches next-step conditional laws, not whole paths, so the distance is sensitive to how information is revealed over time.

The system has three parts. First, a causal temporal encoder, a transformer with causal attention on top of frozen frame features, produces a feature $z_t$ at each step and a prefix classifier. Second, a redundancy stage selects $K$ key steps per clip, with $K$ between 4 and 8. We merge adjacent frames whose features are nearly identical, in the spirit of the cosine-similarity temporal merging of DyCoke, and keep a step when its feature departs from the running history. This step does double duty. It cuts compute, and it coarsens the time axis to the moments where the process actually reveals something. We will measure how much the adapted distance changes under this coarsening and report it, rather than assume it is harmless. The file identifies how pruning alters a filtration as an open question (Gap 5), and this is one of our analyses. Third, the alignment loss is the entropic adapted distance between minibatch source and target key-step processes. We compute it with the Adapted Sinkhorn algorithm of Computational methods for adapted optimal transport, which converges linearly and whose entropic solution converges to the unregularized causal one as the regularization shrinks. Conditional laws are estimated by quantizing each step's features into a few cells per history, which turns a minibatch into a small tree.

That paper notes that exact backward induction is competitive only for very small branching (10 or 25 branches). Compressing to few key steps and few cells per step is what keeps us inside that regime. We then backpropagate through the entropic plan into the encoder.

Stable adversarial training enters as a design lens more than as a component. The Wasserstein GAN argument is that a transport-based discrepancy gives non-vanishing gradients even when the two distributions do not overlap, which binary discriminators do not. Our loss keeps that property but replaces the learned critic, and its gradient penalty and minimax oscillations, with an optimal coupling from a convergent solver. We will test whether this is actually more stable by comparing it against a gradient-penalized critic of the same capacity.

A secondary analysis addresses the disagreement between Temporal Attentive Alignment for Large-Scale Video Domain Adaptation, which up-weights segments where the domain discriminator is confident, and Transferable-Guided Attention Is All You Need for Video Domain Adaptation, which up-weights segments where it is confused. The recursion above assigns a transport cost to each step. We will report whether per-step causal cost correlates with either paper's entropy weights, which gives a principled answer to where alignment effort actually lands.

Baselines

The naive reference point is source-only training of the same causal encoder and prefix classifier with no alignment, evaluated at every observation ratio. Every method must beat it at every ratio to count as adaptation at all.

The strongest existing adversarial method with public code is TA3N (Temporal Attentive Alignment for Large-Scale Video Domain Adaptation, code at TA3N codebase). It aligns multi-scale temporal relations with gradient-reversal discriminators and entropy-based attention, and reports 81.8% on HMDB→UCF. The strongest reported offline number in the file is TranSVAE (Unsupervised Video Domain Adaptation for Action Recognition: A Disentanglement Perspective), with 93.37% average accuracy on UCF-HMDB and 52.6% on Epic-Kitchens, via adversarial alignment of disentangled dynamic factors. Both are clip-level methods. To compare fairly in the streaming regime, we will run each with its alignment objective applied to our causal encoder's prefix features, on the same frozen frame features, with the same number of training steps and the same hyperparameter search budget per method (a fixed number of trials over learning rate and alignment weight, tuned on a held-out source split only). We will also report each at its original offline setting to confirm our reimplementation matches the published numbers. The file gives no reported result for early or prefix-based recognition under domain shift on any of these benchmarks, so the streaming numbers will be new for every method.

Two ablations isolate the key ingredients. Static minibatch Wasserstein, the same Sinkhorn pipeline with the causality constraint removed so that whole key-step sequences are matched as vectors, in the spirit of Optimal Transport for Domain Adaptation, isolates causality. A gradient-penalized Wasserstein critic on flattened sequences isolates "stable transport-based adversary" from "causal transport."

Beating the baselines means the following: higher area under the accuracy-versus-observation-ratio curve than both TA3N and TranSVAE; a statistically significant advantage over static Wasserstein at observation ratios of 30% or less; and no worse than a one-point drop at the full clip. A causal method that wins only at 100% observation would contradict our own mechanism.

Why not just…?

Why not just use ordinary or sliced Wasserstein alignment on the whole clip? This is the most serious threat. Standard optimal transport on complete sequences already sees the joint law of the path, and Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation. is cheap and parallel. The theoretical answer is that path-level Wasserstein metrizes the weak topology, under which sequential decision quantities are unstable (Approximation of martingale couplings on the line in the adapted weak topology). That is an argument, not evidence about video. The static-Wasserstein ablation settles it. If static Wasserstein matches us at early observation ratios on real benchmarks, the causal machinery is not justified for video, and we will say so.

Why not just drop adversarial and transport alignment for temporal contrastive learning, as CoMix does? Contrast and Mix abandons adversarial training as fragile and reaches 90.3% on UCF-HMDB and 64.7% on Jester. However, TranSVAE later surpassed it with adversarial alignment (93.37% versus 90.3% on UCF-HMDB, and 52.6% versus 43.2% on Epic-Kitchens). CoMix's fast/slow speed invariance also deliberately erases timing information, which is exactly what an early predictor needs. We will run CoMix's contrastive objective on our encoder as an additional baseline. If it wins at early ratios, that is a real negative result for our thesis.

Why not just subsample frames uniformly instead of selecting key steps? Uniform sparse sampling works for offline tasks (ClipBERT), but it can skip the brief moments on which early decisions depend, as the key-event views of KeyWorld and Raptis et al. emphasize. We will run uniform $K$-step sampling as an ablation of the redundancy stage, at identical $K$. The expectation is that uniform sampling is competitive at full observation but worse early.

Why not just stabilize the existing discriminator with a gradient penalty? The file notes that no temporal video adaptation method uses Lipschitz-constrained critics, so this simple fix is untested and may recover much of any stability gain. That is why the gradient-penalized critic is a named ablation. If it matches us in stability and accuracy, the contribution reduces to causality alone, and the static-versus-causal comparison must carry the paper.

Proposed Experiments and Decisive Checks

The first check is synthetic and designed so that the two distances disagree. We will build two video domains on the Sprites setup used by TranSVAE. In the source domain, a small motion in the first frames predicts the action. In the target domain, the early frames carry no such cue, but the distribution of complete clips is nearly the same. This is the classic construction where ordinary Wasserstein is small and the adapted distance is large. We will compute both distances exactly on small trees, confirm the disagreement, and check that only causal alignment stops the prefix classifier from relying on the source-only early cue.

On real data, we will evaluate early recognition on UCF-HMDB (both directions), Epic-Kitchens (six transfer tasks), Jester and Kinetics→Gameplay. We report accuracy at observation ratios from 10% to 100% and the area under that curve. We also report training stability as the across-seed standard deviation and the fraction of runs whose target accuracy collapses below source-only, over five seeds. Efficiency is measured as GFLOPs and wall-clock time of the alignment loss, with and without key-step compression. As a sanity reference for compression cost, Efficient Video Transformers with Spatial-Temporal Token Selection reports 33% GFLOP savings at under 0.7% accuracy loss when keeping 50% of tokens on Kinetics-400.

The decisive pattern is an advantage over static Wasserstein that grows as the observation ratio shrinks and disappears at the full clip. If the gap is flat, the gain comes from something other than causality. If the gap is absent, the proposal is refuted for video. We will also report the adapted distance before and after key-step merging, to test whether redundancy compression preserves the information structure the loss is meant to align.