Using Event Silence to Bound Drift in Causal Event-Camera Video Reconstruction

An event camera reports only changes. Each pixel fires a signed spike whenever its log-brightness moves by a fixed contrast step, so the stream records how brightness changes over time but never its absolute level. Networks that turn this stream into video therefore run in a loop: each new frame is built from the incoming events plus the network's own previous output. Small mistakes are fed back and compound, the same way errors compound in autoregressive language models. Over long stretches the reconstructed video slowly brightens, darkens, or picks up ghosts, especially in flat or static regions. Existing remedies either train the network to tolerate its own outputs or smooth its history heuristically. Neither uses the strongest piece of information the sensor gives. When a pixel stays silent, the sensor has measured that pixel's brightness change to be smaller than one contrast step. That measurement holds no matter how long the sequence runs.

We propose turning the event stream into a per-pixel envelope: an interval within which each pixel's log-brightness must lie, relative to the last trusted frame. Fired events shift the interval, and silence keeps it narrow. At every step of the loop, the network's output is projected into this envelope before it is fed back as history. The claim is that this in-loop projection changes how error grows with horizon in a predictable way. In regions where events are silent, error should stop growing and stay below a contrast-step-sized ceiling. In regions where events fire, error should grow only like a random walk in threshold noise, not compound through the feedback loop. Applying the same projection after the fact, without feeding it back, should not achieve this. The aim is a measurable account of which parts of the drift the sensor can correct and which parts only the learned prior can fill in, along with a cheap fix for the correctable part.

Core Claim

Write $L(x,t) = \ln I(x,t)$ for log-intensity at pixel $x$ and time $t$, and $C$ for the contrast threshold. Let $N_\pm(x; t_0, t)$ be the number of positive and negative events at $x$ between an anchor time $t_0$ and $t$. An idealized sensor guarantees that the change since the anchor equals $C(N_+ - N_-)$ up to the sub-threshold residual at each end. Adding a per-event threshold-mismatch noise of standard deviation $\sigma_C$ gives the envelope

$$\big|\,L(x,t) - L(x,t_0) - C\,(N_+ - N_-)\,\big| \;\le\; 2C + k\,\sigma_C\sqrt{N_+ + N_-},$$

where $k$ is a confidence multiplier calibrated on held-out data. The $\sqrt{N}$ term holds if mismatch errors are independent across events, which makes their sum a random walk. Two features of this bound matter. First, for silent pixels ($N_+ + N_- = 0$) the width is a constant $2C$, independent of the horizon. Second, the bound constrains low-frequency, absolute-brightness error, which is exactly what event integration cannot otherwise recover and exactly where the event-reconstruction literature reports drift. E-CIR names brightness drift from accumulated threshold inaccuracies as a core failure. HyperE2VID feeds its previous reconstruction back as context and drifts in sparse-event, slow scenes.

The proposal is to apply a projection (a per-pixel clamp into this interval, softened near the boundary so gradients flow) inside the recurrent loop. The projected frame replaces the raw output as the history the next step conditions on. The network is trained with the projection in the loop, so it learns to fill in what the envelope leaves open (texture, edge sharpness, content inside a wide interval) rather than fighting it.

Relation to Prior Work

Three works motivate the gap. HyperE2VID builds each frame from event voxels and its own previous output, and it treats the resulting train–test mismatch with a curriculum that moves from ground-truth to predicted history. This is the event-camera version of the exposure-bias problem that Zhang et al. characterize for sequence models, but it places no bound on the state at test time. DRFusion confronts autoregressive drift in infrared–visible video diffusion with a low-pass "stabilized history" heuristic. It has no physical measurement between frames, so it has to guess which history to trust. We have such a measurement and use it directly. Sandoval-Segura et al. and Kim argue that error growth in such loops is dynamic and invisible to single-step evaluation. That shapes our evaluation, which always reports error as a function of horizon.

Two works supply method ideas. The Asynchronous Kalman Filter fuses frames and events per pixel and shifts trust between them according to modeled noise. Our envelope is a deliberately cruder, interval-valued cousin of its event model. The difference is where it sits: inside a learned generator's feedback loop rather than as a stand-alone estimator. Fox et al. enforce physical consistency by checking that a reconstruction reproduces the measurements, but they do so through per-sequence optimization lasting hours. We keep the consistency idea and make it a constant-time clamp.

One work is close but different in kind. Lin et al. inject event-derived edges and flow into a video diffusion transformer, but only for interpolation between two real frames. Both ends are anchored, so error cannot propagate. Our setting is causal, with one end open, and that is where propagation happens.

The use of event silence also reframes a finding from the autoregressive literature. Chen et al. show that early decisions in a rollout matter most because everything downstream builds on them. For us, the anchor frame plays that role, and the envelope protects the information it carries for as long as pixels stay silent.

Research Program

The base system is a HyperE2VID-style recurrent reconstructor with two inputs, the event voxel grid and the previous output. We add an optional input, the most recent real frame, for hybrid frame-plus-event sensors. We make three additions.

First, an envelope module tracks $N_\pm$ per pixel since the last anchor. It uses a per-pixel threshold estimate $\hat C(x)$ fitted on calibration sequences. It inflates the bound for known sensor pathologies, namely refractory periods and background-activity noise, using a conformal-style quantile so that a target fraction of ground-truth pixels falls inside the envelope on held-out data.

Second, an in-loop soft projection maps the network's log-output into the envelope before it enters the history buffer.

Third, a training schedule unrolls long sequences with the projection active, so the network sees projected histories.

For event-only sensors, the anchor is the first reconstruction and is re-anchored only when a pixel's envelope grows too wide to be useful. For hybrid sensors, every real frame resets the anchor.

The analysis has two parts. The first splits reconstruction error at each horizon into a part inside the silent-pixel set and a part on firing pixels. The theory predicts a flat curve for the first and roughly $\sqrt{N}$ growth for the second. The second part varies the gap between real frames from zero up to "first frame only." This locates the regime in which the method matters. We expect no advantage when frames are frequent, because each frame already resets drift, and a growing advantage as gaps lengthen.

Baselines

The naive reference is the unmodified recurrent reconstructor trained with plain ground-truth history (teacher forcing), run causally on the same sequences. It shows how fast drift grows when nothing intervenes.

The strongest existing competitor for event-only causal reconstruction is HyperE2VID with its published curriculum, because it is the one method in the file that explicitly targets the error feedback in this loop and runs in real time; it reports 6.6 ms inference with 54% fewer parameters than ET-Net. On accuracy, the file records only that it beats ET-Net and E2VID+ on MVSEC in MSE, SSIM and LPIPS, with no absolute values. For causal long-horizon error curves, no reported result was found on Lacuna for any method. We will therefore rerun HyperE2VID from its released configuration on identical sequences and horizons, using the same training data, the same number of GPU-hours, and the same hyperparameter-search budget as our variant. Our method is built on the same backbone, so the comparison isolates the envelope.

For hybrid frame-plus-event input, the strongest classical competitor is the Asynchronous Kalman Filter, which reports a 48% MSE reduction and 11% SSIM gain over event-only E2VID on CED. The file gives these as relative figures and no absolute MSE. We run it with its published noise parameters, re-tuned on our calibration split.

A further baseline combines the two: the HyperE2VID output post-filtered by the Kalman filter. This is the obvious "learned prior plus physics" pipeline that does not feed the filtered result back into the loop.

The key ablation applies the identical envelope projection only to the displayed output and never to the history. If this matches the in-loop version, the effect is ordinary post-processing, and the claim about error propagation fails. A second ablation replaces the inference-time projection with an envelope-violation penalty during training only.

To beat the field, the in-loop method must achieve two things over HyperE2VID and over the post-filtered pipeline. It must lower the slope of mean absolute log-intensity error against horizon on silent pixels to near zero. It must also improve LPIPS at long horizons (several seconds of event-only rollout) without losing more than a negligible amount at short horizons. For context only, non-causal deblurring reports 0.114 MSE for E-CIR and 29.69 dB PSNR / 0.9039 SSIM for Fox et al. on REDS. These are short-window figures and are not directly comparable.

Why not just…?

Why not just use better curriculum or noise-augmented history during training? This is what HyperE2VID already does. Training on one's own outputs widens the set of histories the network can handle, but it places no ceiling on how far a test-time rollout can wander. Sandoval-Segura et al. and Kim both find that bounded per-step training error does not prevent long-horizon compounding. The baseline comparison tests this directly. If a strong curriculum alone flattens the silent-pixel error curve, our mechanism is unnecessary, and we will report that outcome.

Why not just run a Kalman filter on the network's output? This is the most serious alternative, because the filter already encodes the event physics. Its weakness is that it treats each pixel as an independent linear estimate, and a post-hoc filter cannot stop the network from conditioning on its own drifted history. The output-only ablation and the post-filtered pipeline settle this question. If they match the in-loop version, the contribution reduces to "filter your outputs," and we would say so.

Why not just add an event-consistency loss? A training-time penalty teaches the network to usually respect the envelope on training-like data. It guarantees nothing once the history drifts out of distribution, which is the regime of interest. The loss-only ablation measures how much of the gain survives without the inference-time projection.

Why not just interpolate between real frames, or re-anchor on every frame? When frames arrive often, re-anchoring alone bounds drift, and we expect our method to add little there. Lin et al. show that two-ended interpolation works well, reaching 8.168 FID on BS-ERGB. That approach must wait for the future frame, so it is unavailable to causal, low-latency use and to frame dropouts. We include it as a non-causal upper reference, and the frame-gap sweep shows where the causal gap opens.

Proposed Experiments

Event-only experiments use ECD, HQF and MVSEC, the sequences on which HyperE2VID was evaluated. They are given one initial reconstruction and then run causally for the full sequence. Hybrid experiments use CED, which has real color frames and events across a wide lux range, and BS-ERGB, with frames withheld at gaps of 1, 4, 16, 64 frames and "first frame only." REDS with simulated events provides controlled studies in which $C$ and the mismatch noise are known exactly. This lets us check whether the predicted $\sqrt{N}$ growth on firing pixels and the flat curve on silent pixels appear when the noise model is correct, before moving to real sensors where it is only approximate.

Metrics, all reported as curves over horizon, are MSE, SSIM and LPIPS. We also report three drift-specific measures: mean signed log-intensity error on silent pixels, the fraction of ground-truth pixels inside the calibrated envelope (to verify calibration), and the rate at which unprojected network outputs violate the envelope. Runtime is measured on the same GPU as HyperE2VID's 6.6 ms figure; the projection adds per-pixel counting and clamping only.

The decisive check is the in-loop versus output-only comparison at long horizons on real data. The theory predicts that only the in-loop version flattens the silent-pixel error curve. If both versions give the same curve, the error-propagation story is wrong.

A second check targets real sensor noise. On CED's low-light sequences, background-activity events inflate $N$, so the envelope should widen and the advantage should shrink. If instead real noise makes the calibrated envelope routinely exclude the truth, so that projection hurts, the idealized sensor model is too optimistic for practical use, and that negative result is itself worth reporting.