Abstract
HyperE2VID introduces a dynamic neural network architecture designed to reconstruct high-quality video from event-based camera data. By leveraging hypernetworks to generate "tailor-made" convolutional filters for every pixel in real-time, the model adapts to the highly variable nature of event streams. HyperE2VID achieves state-of-the-art image quality while significantly reducing memory and computational requirements compared to previous heavy-weight models, making it suitable for high-speed robotic applications.
Introduction
Traditional frame-based cameras capture snapshots at fixed intervals, often resulting in motion blur or poor dynamic range in challenging lighting. Event cameras, or neuromorphic sensors, address these issues by asynchronously reporting per-pixel brightness changes, offering low latency and high dynamic range. However, reconstructing standard intensity images from sparse, non-uniformly distributed event data is difficult. Static neural networks with fixed weights struggle to adapt to varying event densities, often producing artifacts like blur or low contrast when scene dynamics shift.
Methodology
HyperE2VID addresses the limitations of static networks by employing Hypernetworks to implement Dynamic Convolutions. The architecture consists of three primary components:
- Context Fusion (CF) Module: This module integrates the incoming event stream (capturing motion) with the previously reconstructed frame (capturing static structure) into a "context tensor."
- Dynamic Filter Generation (DFG): A hypernetwork uses the context tensor to predict unique convolutional kernels for every pixel at each time step. This allows the network to adapt its processing based on local scene dynamics, such as fast-moving objects versus static backgrounds.
- Efficiency via Decomposition: To manage the computational cost of per-pixel filter generation, the authors utilize Fourier-Bessel bases. The hypernetwork predicts a small set of compositional coefficients to combine these pre-defined bases, maintaining flexibility while drastically reducing mathematical overhead.
The model is trained using a Curriculum Learning strategy, where it initially utilizes ground-truth frames for context before transitioning to its own previous reconstructions, ensuring stable convergence of the recurrent architecture.
Experimental Results
The model was evaluated on five major real-world benchmarks: ECD, MVSEC, HQF, FPVDR, and CED. These datasets cover diverse scenarios, including high-speed drone racing and night-driving sequences.
HyperE2VID sets a new state-of-the-art (SOTA) for event-based reconstruction:
- Superior Quality: It outperforms existing methods like ET-Net and E2VID+ across MSE, SSIM, and LPIPS metrics.
- Efficiency: The model uses approximately 54% fewer parameters and 44% fewer FLOPs than the previous leader, ET-Net.
- Real-time Performance: With an inference time of 6.6ms on standard GPUs, it is viable for real-time applications.
Conclusion and Future Work
HyperE2VID demonstrates that spatial adaptation via hypernetworks is more effective than simply increasing network depth or width. While the current model uses channel-wise concatenation for context fusion, future work could explore more sophisticated attention mechanisms. Additionally, the hyper-architecture approach holds promise for other event-based tasks like optical flow estimation and semantic segmentation, which also suffer from data sparsity challenges.
Summary updated