Current systems that reconstruct video from brain recordings produce a two-second clip, stop, and start over. Long-video generators that roll forward frame by frame have the opposite problem: they run for minutes, but nothing outside the model keeps them tied to reality, so small errors pile up. We propose to join the two. A causal video generator rolls forward continuously, and a live electroencephalography stream (EEG, scalp electrical recordings) steers every frame. The central question is whether the brain signal keeps its grip on the generated video as time goes on, or whether the generator's learned sense of what videos usually look like gradually takes over. We expect the answer to depend on how the brain signal enters the model. If aligned EEG features are mixed into each frame's noisy input before denoising starts, rather than attached late as a side prompt, we expect the brain signal to stay in control. We expect the gap between the two designs to be largest for people the model never saw during training, because their signal is noisiest.
The contribution has two parts. The first is a measurement. We will plot fidelity to the viewed video as a function of time into the rollout, together with a "switch test" that asks how quickly the generation follows when the viewer's stimulus changes. No existing brain-to-video paper reports either quantity. The second is a method that keeps this curve flat for unseen subjects. Three ingredients do the work. Each subject's recording is statistically re-centered without any labels. Each frame's EEG features are fused into that frame's noisy input tokens early in the network. During training, the model's own past frames are deliberately corrupted with noise, so it cannot ignore the brain and simply extrapolate from its history.
Why this gap is real
Every brain-to-video system in the file decodes fixed short windows. EEG2Video introduced the SEED-DV benchmark and maps 2-second EEG windows to 16 frames. It adjusts the starting noise according to a predicted fast or slow motion label. DynaMind adds region-wise EEG encoders and a "motion blueprint" that shapes the initial noise. NEED adds a pre-trained module that normalizes individual differences, and it retains 93.7% of within-subject classification accuracy on unseen subjects. MindCross separates shared encoders from subject-specific ones for fast calibration. CineSync fuses EEG and functional MRI (fMRI) to condition a large video diffusion model, but only within-subject. All of these reset at every window, so none of them can show horizon decay, whether as drift or as a loss of responsiveness to the stimulus.
The long-horizon generators take the opposite approach. Causal Forcing and AAD-1 distill bidirectional video models into causal, few-step, real-time generators. DynaMem and LIVE add memory so a scene survives long rollouts. All four condition on text, actions, or pixels. The file's own synthesis states the tension directly: pre-trained video priors tend to overpower noisy neural conditioning. The file also points to GLFA as evidence, since it reaches 84.7% cross-subject semantic accuracy while its SSIM (a pixel-structure similarity score) stays at 0.17–0.19. Nobody has asked what happens to that tension over time. The closest intersection work, NEED, links cross-subject decoding to video generation, but it keeps the 2-second window and relies on frozen high-level embeddings. Our proposal differs in three ways: it is a streaming rollout, it measures fidelity as a function of horizon, and the brain signal enters every frame's input rather than a clip-level prompt.
Core Claim
In a causal video generator, the influence of a conditioning stream decays with horizon at a rate set by two things. One is how much the model can rely on its own history instead of the condition. The other is how noisy the condition is for this particular person. Late cross-attention conditioning lets the model lean on history, which is the cheapest way to reduce denoising error once a scene is established, so neural influence fades. Unseen subjects make this worse because their features are further from the training distribution.
We propose two remedies that change the model's incentives. Early per-frame fusion puts the brain signal inside the tokens being denoised. Diffusion-Forcing-style history corruption during training makes history an unreliable shortcut. Diffusion Forcing supplies the technique of training with independent noise levels per frame, and we reuse it here for a new purpose: the noise level applied to history becomes a dial between temporal consistency and neural anchoring. The testable prediction is specific. As history noise increases, cross-window consistency should fall slightly while the switch test and late-horizon fidelity improve, and the improvement should be larger for held-out subjects than for training subjects.
Research Program
The EEG encoder starts from LaBraM via its released code (LaBraM codebase). AVDE used this same encoder for static images, but only with spatial-scale autoregression.
Before encoding, each subject's raw EEG is re-centered with Euclidean Alignment. This method, which Rodrigues et al. combine with augmentation, whitens each subject's signal by that subject's average channel covariance, and it needs only unlabeled recordings from the new person. We treat it as a cheap, calibration-free step. As a comparison, we also include per-subject batch-norm statistics in the style of the SPD domain adaptation in TSMNet. The file flags a real risk here (Gap 3): covariance centering may remove phase information within short windows. We therefore keep the time-domain patches and apply the whitening as a linear map on channels, which preserves temporal structure.
The generator is a causal latent video model distilled following the Causal Forcing recipe, at the 8 fps and modest resolution used by the benchmarks. At each frame, EEG tokens from a sliding window ending at that frame are projected and added to, or concatenated with, the frame's noisy latent tokens at the first transformer block. This is early fusion. The late-fusion variant instead feeds the same tokens only through cross-attention. CineSync found that raw-concatenation "joint" encoders underperform modality-specific towers followed by a learned projection. We follow that result and fuse projected EEG tokens, never raw signals.
Training uses rollout-style objectives, as in HERO, so the model sees its own drift during training. On top of that, history frames receive random per-frame noise levels. The semantic objective aligns EEG to CLIP text embeddings, because DynaMind found text-embedding alignment more stable than image-embedding alignment.
The analysis centers on three measures. The first is the horizon-resolved fidelity curve: SSIM and CLIP similarity between generated and viewed frames, binned by seconds since rollout start. The second is switch latency: the number of frames until the generation's semantics match the new stimulus after a scene cut. The third is a counterfactual swap: replace the EEG from time $t$ onward with EEG from an unrelated segment and measure how far the generation diverges. A model that ignores the brain will not diverge.
Baselines
The naive reference is a generator given no EEG after the first window. It is prompted once and rolls out unconditionally. Its fidelity curve shows what the video prior alone produces over time, and any brain-conditioned model must stay clearly above it at late horizons. It also calibrates the swap test: this model diverges by zero.
The strongest existing competitor for fidelity is independent windowed decoding. We run NEED-style or EEG2Video-style decoding on each consecutive 2-second window and concatenate the clips. Because it resets at every window, it has no horizon decay by construction, which makes it the hardest bar on the fidelity curve. On SEED-DV, EEG2Video reports 79.8% 2-way semantic accuracy and SSIM 0.256 on the 40-class set. DynaMind reports a 9.4% relative SSIM gain and a 19.7% relative reduction in Fréchet Video Motion Distance over EEG2Video. NEED reports 92.4% cross-subject retention of reconstruction quality. On CineBrain, CineSync reports 2-way semantic accuracy 0.926 and Fréchet Video Distance (FVD) 44.77, within-subject. We found no result on Lacuna for any method on horizon-resolved fidelity, switch latency, or cross-window consistency for brain-conditioned video.
For fairness, we reimplement the windowed baseline with the same LaBraM encoder, Euclidean Alignment, generator backbone, data splits, and training compute. The only difference is that it resets per window. Beating it means matching its per-window SSIM and 2-way accuracy within noise while clearly improving cross-window consistency, measured by FVD over full sequences and by identity or background persistence within a shot, without worse switch latency.
The second strong competitor isolates our key ingredient. It is the same causal generator with EEG supplied only through late cross-attention, trained with the same rollout objective but with clean history. We also run the two crossed ablations: early fusion without history corruption, and late fusion with it. These tell us whether early fusion, history noise, or their combination produces any flattening of the curve. For cross-subject calibration, we compare against a MindCross-style subject-specific adapter trained on a few minutes of the held-out subject's labeled data. MindCross reports that its adapter updates 9.77M parameters, versus more than 100M for full retraining. Its adapter receives the same encoder and generator, and we give it the tuning budget its paper describes.
Why not just…?
Why not just decode each 2-second window independently and stitch the clips together? This is the most serious threat. If stitched windows match our model's late-horizon fidelity and people or reviewers judge cut-to-cut seams acceptable, then rollout adds little. It is plausible that stitching wins on raw per-window SSIM, since a reset can never drift. What stitching cannot provide is continuity inside a shot. CineBrain's episodes contain long shots in which the same characters and set persist, and independently decoded windows will re-sample identity and layout every two seconds. The experiment settles this directly. If our model does not improve full-sequence FVD and within-shot identity persistence at equal per-window fidelity, the rollout idea fails and we will report that.
Why not just apply Euclidean Alignment to an existing decoder and skip the new architecture? Alignment addresses who is being decoded, not when. It should improve the level of the fidelity curve for unseen subjects, but it gives the causal generator no reason to keep attending to EEG once a scene is established. We run alignment in every arm, including the late-fusion causal model. If late fusion with alignment already has a flat curve on held-out subjects, our fusion and history-noise claim is wrong.
Why not just use late cross-attention conditioning, which is how text-to-video models are already steered? Late conditioning is standard and works for text, which is a clean signal. Our hypothesis is that it fails specifically when the condition is noisy, because the cheapest route to low denoising loss is to copy history. The crossed ablation tests this. The prediction is not "early is better" in general. The prediction is that the early-versus-late gap should grow with horizon and should be larger on held-out subjects than on training subjects. If the gap is constant over time, fusion position is a static quality factor and has nothing to do with horizon decay.
Why not just calibrate a small adapter per new user? Calibration is realistic and may close most of the cross-subject gap. It is not a competitor to anchoring, because an adapted model can still drift away from the EEG over time. We include it as a baseline and also combine it with our model. The cross-subject part of our claim survives only if the uncalibrated anchored model approaches the calibrated late-fusion model at late horizons.
Proposed Experiments
The primary testbed is CineBrain, because its 540 minutes of continuous television viewing is the only dataset in the file with long, naturally continuous stimuli and synchronized EEG. We use its EEG only, with leave-one-subject-out evaluation across its 6 participants. The small cohort is a real limitation: NEED reports that generalization jumps once training reaches 10–15 subjects. We therefore add SEED-DV, which has 20 subjects, as a cross-subject stress test. There we chain 2-second clips in presentation order into 20–30 second sequences whose hard category cuts provide many switch events. Rollouts run for 10, 30, and 60 seconds on CineBrain.
The metrics are per-horizon SSIM, CLIP similarity, and 2-way semantic accuracy against the viewed frames. We add full-sequence FVD, switch latency at annotated cuts, swap-test divergence, and within-shot identity persistence measured by face or object embeddings. We report within-subject and held-out-subject curves separately, and every comparison uses matched compute.
Three results would decide the question. The first is whether the late-fusion causal model's held-out-subject fidelity curve falls faster than its within-subject curve; this is the existence of subject-dependent decay. The second is whether early fusion with history noise flattens that curve relative to the late-fusion model at 30 and 60 seconds, while matching the stitched-window baseline's per-window fidelity and beating it on full-sequence FVD. The third is whether sweeping the history-noise level traces the predicted trade-off, with consistency falling and switch latency improving. If the first result does not appear, the premise is wrong and the paper becomes a measurement report. If the second does not appear, stitching is the recommended practice.
References
- EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals
- DynaMind: Reconstructing Dynamic Visual Scenes from EEG by Aligning Temporal Dynamics and Multimodal Semantics to Guided Diffusion
- NEED: Cross-Subject and Cross-Task Generalization for Video and Image Reconstruction from EEG Signals
- MindCross: Fast New Subject Adaptation with Limited Data for Cross-subject Video Reconstruction from Brain Signals
- CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processing
- Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation
- AAD-1: Asymmetric Adversarial Distillation for One-Step Autoregressive Video Generation
- DynaMem: Consistent Long Video Generation via Hierarchical Memory and Motion Priors
- LIVE: Long-horizon Interactive Video World Modeling
- Enhancing Cross-Subject fMRI-to-Video Decoding with Global-Local Functional Alignment
- Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion
- Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI
- Autoregressive Visual Decoding from EEG Signals
- Combining Euclidean Alignment and Data Augmentation for BCI decoding
- HERO: History-Enriched Rollout Training for Long-Horizon Autoregressive Neural Operators