HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworks

Burak Ercan, Onur Eker, Canberk Saglam, Aykut Erdem, Erkut Erdem

TL;DR

HyperE2VID introduces a dynamic neural network that reconstructs high-quality video from event-based cameras by generating "tailor-made" convolutional filters for every pixel in real-time. By using hypernetworks to adapt to the highly variable nature of event data, it achieves state-of-the-art image quality while requiring significantly less memory and computational power than previous heavy-weight models.

What Problem Does This Paper Address?

Traditional cameras capture the world in "frames"—snapshots taken at fixed intervals. This leads to motion blur in high-speed scenes and "blown-out" highlights in high-contrast environments. Event cameras (or neuromorphic sensors) work differently: they only report changes in brightness for each pixel asynchronously. This allows for incredibly low latency and a massive dynamic range.

However, turning these sparse "events" back into a standard video (intensity images) is difficult. Because the density of event data changes wildly depending on how fast things are moving, standard "static" neural networks—where the weights are fixed after training—struggle to adapt. They often produce reconstructions with blur, low contrast, or "smearing" artifacts when the scene dynamics shift.

Methods

The authors propose HyperE2VID, which moves away from the "one-size-fits-all" approach of static networks. Its core innovation is the use of Hypernetworks to implement Dynamic Convolutions.

  1. Context Fusion (CF) Module: The network looks at two things to understand the current scene: the incoming stream of events (which captures motion) and the previously reconstructed frame (which captures static structure). It fuses these into a "context tensor."
  2. Dynamic Filter Generation (DFG): Instead of using pre-set weights, a hypernetwork uses that context tensor to predict a unique convolutional kernel for every single pixel at every time step. This allows the network to use different "lenses" for fast-moving objects versus static backgrounds.
  3. Efficiency via Decomposition: Generating a full 5x5 filter for every pixel would be computationally ruinous. To solve this, the authors use Fourier-Bessel bases. The hypernetwork only predicts a few "compositional coefficients" that combine these pre-defined bases, drastically reducing the math required while maintaining flexibility.
  4. Curriculum Learning: Training a recurrent network that relies on its own previous (and initially poor) reconstructions is hard. The authors used a strategy where the network initially "cheats" by looking at ground-truth frames during training, gradually transitioning to its own predictions as it gets smarter.

Datasets

The model was evaluated across five major real-world benchmarks to ensure it wasn't just "overfitting" to synthetic data:

  • ECD (Event Camera Dataset): The standard for 6-DOF camera motion.
  • MVSEC (Multi Vehicle Stereo Event Camera): Includes challenging night-driving sequences.
  • HQF (High-Quality Frames): Focused on well-exposed, sharp ground truth.
  • FPVDR (UZH-FPV Drone Racing): Extreme, high-speed motion.
  • CED (Color Event Camera Dataset): Used to demonstrate the model's ability to reconstruct color video.

Key Insights and Ideas

  • Events and Frames are Complementary: The "Aha!" moment here is that events are great for dynamic edges but terrible for static textures, while intensity images are the opposite. By using both as context for filter generation, the network knows exactly when to rely on which data source.
  • Spatial Adaptation is Key: In a single scene, one corner might be a blurry mess of high-speed motion while the other is a still wall. HyperE2VID handles this by applying different math to different parts of the same image simultaneously.
  • Dynamic > Deep: The results suggest that making a network adaptive is more effective than simply making it deeper or wider. HyperE2VID beats models with twice as many parameters by being smarter about how it uses the parameters it has.

Key Results

HyperE2VID sets a new benchmark for event-based reconstruction:

  • Superior Quality: It outperformed existing SOTA methods like ET-Net and E2VID+ across almost all metrics (MSE, SSIM, and LPIPS).
  • Efficiency: Despite the "dynamic" overhead, it uses ~54% fewer parameters and ~44% fewer FLOPs (Floating Point Operations) than the previous leader, ET-Net.
  • Speed: It achieves an inference time of 6.6ms on a standard GPU, making it viable for high-speed robotic applications.
  • Robustness: Qualitative results show it maintains sharp edges and high contrast even when generating video at 5000 FPS or during "still" periods where event data drops to near zero.

Limitations and Future Work

While HyperE2VID is a major step forward, the authors note a few areas for growth:

  • Context Fusion Complexity: The current fusion of events and images is relatively simple (channel-wise concatenation). More sophisticated attention mechanisms might yield even better results.
  • Task Expansion: The "Hyper-architecture" approach could likely be applied to other event-based tasks, such as optical flow estimation or semantic segmentation, which also suffer from data sparsity.
  • Post-Processing: The paper mentions an optional post-processing step to clean up textureless regions; integrating this directly into the end-to-end learning process could further streamline the pipeline.

Summary updated

Abstract

Event-based cameras are becoming increasingly popular for their ability to capture high-speed motion with low latency and high dynamic range. However, generating videos from events remains challenging due to the highly sparse and varying nature of event data. To address this, in this study, we propose HyperE2VID, a dynamic neural network architecture for event-based video reconstruction. Our approach uses hypernetworks to generate per-pixel adaptive filters guided by a context fusion module that combines information from event voxel grids and previously reconstructed intensity images. We also employ a curriculum learning strategy to train the network more robustly. Our comprehensive experimental evaluations across various benchmark datasets reveal that HyperE2VID not only surpasses current state-of-the-art methods in terms of reconstruction quality but also achieves this with fewer parameters, reduced computational requirements, and accelerated inference times.
BibTeX
@article{ercan2024hypere,
  author = {Burak Ercan and Onur Eker and Canberk Saglam and Aykut Erdem and Erkut Erdem},
  title = {{HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworks}},
  journal = {IEEE Trans. Image Process.},
  year = {2024},
  doi = {10.1109/tip.2024.3372460},
}
Figure