Continuous Video Reconstruction via Asynchronous Event Fusion

36 concepts27 papers1 proposal

Traditional cameras capture the world as a sequence of discrete, static frames. While this paradigm is intuitive, it imposes fundamental physical limitations. During rapid motion, standard sensors suffer from two distinct failure modes: temporal blind spots (the gaps between consecutive frames) and motion blur (the smearing of light that occurs when an object moves while the shutter is open).

To overcome these sampling limits, a rapidly growing body of research explores the fusion of standard frame-based cameras with neuromorphic event sensors. Instead of capturing absolute intensity at fixed intervals, event cameras record asynchronous, microsecond-level changes in log-intensity. As surveyed by Bharatesh Chakravarthi et al., this paradigm shifts the problem from estimating motion blindly to leveraging a continuous, physically grounded map of spatiotemporal changes.

The Intuition: Events as Temporal Derivatives

Why does fusing event data with standard frames work so well? Mathematically, an event camera acts as a continuous-time differentiator. When the log-intensity of light at a specific pixel changes past a set threshold, the sensor emits an asynchronous event containing a precise timestamp, pixel location, and polarity (signifying brightening or darkening).

$$\Delta \ln I(x, y, t) \approx \pm C$$

If a system captures a blurry frame or encounters a large temporal gap between two sharp frames, the event stream provides the exact temporal derivative of the scene's intensity during those intervals. By integrating these derivatives over time, models can reconstruct absolute intensity values at any arbitrary timestamp.

As demonstrated by Chen Song et al. in their E-CIR framework, representing video as a continuous parametric function driven by integrated event data yields a massive leap in reconstruction accuracy compared to predicting discrete states. To handle the inherent noise and uncertainties of both sensors—such as overexposure in frames and refractory noise in events—the authors of another approach introduce an Asynchronous Kalman Filter (Ziwei Wang et al.) to maintain a sharp, continuous-time intensity guide that prevents ghosting artifacts.

Undoing the Smear: Event-Guided Motion Deblurring

Motion blur occurs because a standard pixel acts as an integrator over an exposure window $T$. If an object moves, its intensity is averaged across multiple spatial coordinates. However, if we possess a microsecond-level record of all changes within $T$, we can mathematically reverse this integration.

Frame Camera:   [------- Integrated Exposure Window (Blur) -------]
Event Camera:   --•---•--••---•----•---•••--•----•--•---•--•---> (Continuous Derivatives)
Reconstructed:  [Sharp t1] -> [Sharp t2] -> [Sharp t3] -> [Sharp t4]

Several architectures exploit this to restore sharp details from degraded inputs:

  • Structural Priors: Rather than treating events merely as extra pixels, modern approaches use them as a structural skeleton. K. Liu et al. employ cross-modal attention mechanisms to anchor sharp textures precisely where events indicate motion occurred.
  • Asymmetric Processing: Jinjian Wu designs an asymmetric interaction network that specifically transfers fine-grained spatial details from the event stream to sharpen low-level image edges, while using the image branch to enrich high-level event semantics.
  • Direct Texture Mining: Dachun Kai shifts the utility of event sensors from purely motion estimation to direct high-frequency detail reconstruction, mining textures directly from the event stream to enhance low-resolution video.
  • Cross-Modal Deconvolution: This principle extends beyond visible light. Lin Zhu et al. demonstrate that high-speed event streams can act as a temporal map for deconvolution to restore sharp thermal details to blurry infrared frames. Even for alternative sensor modalities like photon-counting SPADs, Manasi Muglikar et al. show that events provide a continuous motion reference to resolve spatial details from aggregated photon counts in near-darkness.

By filling in the missing information between the start and end of an exposure, these methods effectively reverse motion blur (Lei YU et al.) and provide a nearly continuous record of non-uniform motion (Hongguang Zhang et al.).

Bridging the Gaps: Video Frame Interpolation and Alignment

In traditional Video Frame Interpolation (VFI) and Video Super-Resolution (VSR), the primary bottleneck is alignment. If an object moves non-linearly or undergoes complex deformations between frame $A$ and frame $B$, standard optical flow algorithms fail because the pixel displacement is too large to match accurately.

Event streams solve this by acting as an asynchronous temporal bridge across the sensor's blind time. Yongcheng Jing et al. synthesize high-frequency intermediate frames from event streams, reducing the pixel displacement between consecutive steps and simplifying spatial alignment. Instead of relying solely on events for flow, Jiaben Chen et al. use high-temporal-resolution events as a corrective signal to refine initial RGB-based optical flow estimates, mitigating event sparsity and noise while maintaining spatial structure.

For highly chaotic or non-linear motion, traditional warping fails entirely. To bypass this:

  • Direct Synthesis: Yongjian Deng et al. propose a "direct synthesis" strategy, transforming raw event streams into a grayscale-like structural map at the exact timestamp of the missing frame to capture complex geometry without explicit warping.
  • Timestamp-Aware Interpolation: This concept is pushed to extreme scales by Bo Zhang et al., who use real-time event data to guide intermediate frame generation, reconstructing high-fidelity video at 10,000 FPS from a standard compressive camera.
  • Tracking Through Degradation: Haojie Yan et al. demonstrate that tracking non-linear trajectories via events bridges the blind time of traditional sensors, preventing ghosting artifacts.

This cross-modal alignment is critical when image degradation is severe. Hoonhee Cho et al. show that the event stream's immunity to motion blur allows systems to establish temporal correspondences across frames that are otherwise unrecognizable. This capability is further validated by benchmarks like BlinkVision, introduced by Xiaoyu Shi et al., which evaluates how continuous brightness changes track motion during exposure failures, and extended to 3D tracking tasks like joint optical and scene flow estimation (Zhexiong Wan et al.). Even in state estimation tasks like monocular visual odometry, Guillermo Gallego et al. demonstrate that tracking motion during the sensor's blind time enables accurate state estimation during high-speed movements.

From Pixels to Continuous Fields: Implicit Representations and 3D Scenes

The ultimate realization of this paradigm moves away from discrete pixel grids entirely, modeling scenes as continuous spatiotemporal representations. By leveraging Implicit Neural Representations (INRs), frameworks can reconstruct video at arbitrary spatial scales and frame rates.

Yunfan LU et al. combine asynchronous events with standard RGB frames within an INR framework to achieve video super-resolution at arbitrary, non-integer scales. The authors extend this unified approach in UniINR (Yunfan LU et al.), simultaneously correcting rolling shutter distortion, removing motion blur, and interpolating frames by mapping pixel-level movement trajectories. Because events capture the underlying physics of motion as a continuous signal, Feng Li et al. demonstrate that event-guided models can generalize to extreme upsampling scales far beyond their training distribution.

[Discrete Frames] + [Asynchronous Events] 
       │
       ▼
[Implicit Neural Representation (INR)] 
       │
       ├─► Arbitrary Spatial Scale (Super-Resolution)
       ├─► Arbitrary Temporal Scale (Continuous Frame Interpolation)
       └─► Distortion Correction (Rolling Shutter / Blur Removal)

This continuous temporal bridging has also revolutionized 3D scene reconstruction. In low-light or high-speed scenarios, traditional Neural Radiance Fields (NeRF) fail due to motion blur. Lin Zhu et al. (E2NeRF) solve this by using asynchronous brightness changes as a temporal constraint, aligning predicted 3D transitions with physical events recorded during exposure.

Similarly, for free-trajectory video captures, Bohao Liao integrates event cameras into 3D Gaussian Splatting (EF-3DGS). By using the continuous event stream to bridge discrete video frames, the system recovers precise camera trajectories and scene details during the blind intervals of rapid camera movements, ensuring stable 3D reconstruction where frame-only methods fail.

Key Foundational Work

Turning Frequency to Resolution: Video Super-Resolution via Event Cameras (2021) — contributed 3 concept elements to this direction.
Event-aided Direct Sparse Odometry (2022) — contributed 1 concept elements to this direction.
An Asynchronous Kalman Filter for Hybrid Event Cameras (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_event_guided_motion_reconstruction_f4a40eb6,
  title = {event-guided motion reconstruction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/event-guided-motion-reconstruction/rdr_f4a40eb68fee4f25843f403821790153},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_event_guided_motion_reconstruction_f4a40eb6_v1,
  title = {event-guided motion reconstruction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/event-guided-motion-reconstruction/rdr_f4a40eb68fee4f25843f403821790153/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake