Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras

Ruiming Cao, Dekel Galor, Amit P. S. Kohli, Jacob L. Yates, Laura Waller

TL;DR

Event cameras are traditionally "blind" to static scenes because they only respond to changes in light intensity. Noise2Image flips this limitation on its head by proving that the "noise" events typically filtered out as background activity are actually correlated with static scene intensity. By modeling these photon-noise statistics, the authors can reconstruct high-quality static images from an event stream without any additional hardware.

What Problem Does This Paper Address?

Event cameras (also known as Neuromorphic cameras or Dynamic Vision Sensors) operate differently than standard CMOS sensors. Instead of capturing frames at fixed intervals, each pixel independently reports when it detects a change in log-intensity. This makes them exceptional for high-speed tracking and high dynamic range (HDR) imaging, but it leaves them unable to "see" anything that isn't moving.

To get a static background image, researchers usually have to add a second conventional camera, use complex opto-mechanical prisms to "shake" the view, or rely on the camera itself being in constant motion. The authors of this paper identified a hidden signal in the "background activity"—noise events triggered by random photon arrivals—and sought to use this noise to recover the invisible static scene.

Methods

The core of the paper is a transition from treating noise as a nuisance to treating it as a measurement.

  1. Statistical Modeling: The authors derived a theoretical model for noise event probability ($p_e$). They observed that because event cameras have logarithmic sensitivity, the variance of the log-intensity is higher in low-light conditions. This means random photon fluctuations are more likely to cross the contrast threshold and trigger a "noise event" in darker areas than in brighter ones.
  2. Accounting for Hardware: Real-world event circuits include bias voltages to stabilize readings. The authors added a photoreceptor bias term ($b_{pr}$) to their model, which explains why noise event rates actually peak at a certain low-light level and then decrease as it gets even darker (a non-monotonic relationship).
  3. Noise2Image Architecture: They trained a modified U-Net to perform the inverse mapping: taking a 2D matrix of aggregated noise event counts and outputting an intensity image.
  4. Polarity Utilization: Crucially, the network uses two input channels—one for positive events and one for negative events. Because hardware leakage often makes positive events easier to trigger, the two polarities provide complementary information that helps resolve ambiguities in the intensity-to-noise mapping.
Overview of the Noise2Image pipeline, showing how noise events are aggregated and mapped to a static recovery image.

Datasets

The authors introduced a new dataset called NE2I (Noise Events-to-Image) because existing event camera datasets focus almost exclusively on motion.

  • Training/Validation: 1,004 high-resolution artistic portraits from Unsplash, displayed on a calibrated LCD monitor and recorded with a Prophesee Metavision EVK3-HD camera.
  • Out-of-Distribution Testing: 100 images from the DIV2K super-resolution dataset to test how well the model generalizes beyond human faces.
  • Real-World Data: The authors also tested the model on live indoor and outdoor scenes (e.g., a clock tower) to prove it works outside of a controlled lab environment.

Key Insights and Ideas

  • Noise is a Signal: The most profound insight is that "background activity" in event cameras isn't just random; it's a function of the underlying static illuminance.
  • Logarithmic Sensitivity is Key: The reason this works for event cameras and not standard cameras is the log-intensity trigger. It amplifies the relative fluctuations of photon noise in a way that correlates directly with the scene.
  • Resolving Ambiguity with Polarities: The relationship between intensity and noise rate is one-to-many (non-monotonic). However, by looking at the ratio and counts of positive vs. negative events, the neural network can "triangulate" the correct intensity value.
Fig. 2. a. Theoretical noise event probability, 𝑝𝑒, versus the average photon count, 𝜆, at different photoreceptor bias values, 𝑏𝑝𝑟(𝜖is the contrast threshold). b. Experimentally measured noise event rate versus illuminance (proportional to 𝜆) matches well with our theoretical model after fitting parameters 𝜖, 𝑏𝑝𝑟and 𝑁.

Key Results

  • Static Recovery: Noise2Image achieved a PSNR of 25.0 dB on in-distribution data, vastly outperforming standard event-to-video (E2VID) methods which hover around 7-9 dB because they expect motion to generate signal.
  • Generalization: Even on out-of-distribution scenes (DIV2K), the model maintained a PSNR of 19.3 dB, showing it learns local correlations rather than just memorizing image types.
  • Hybrid Scenes: The authors demonstrated that they could separate "signal" events (from a moving fan) from "noise" events (from the background), using E2VID for the fan and Noise2Image for the background, then stitching them together for a complete reconstruction.
Qualitative comparison showing Noise2Image recovering clear facial features and architectural details from noise counts that look like static to the naked eye.

Limitations and Future Work

  • Brightness Regimes: The method works best in low-to-moderate light. In very high brightness (e.g., direct sunlight), photon noise fluctuations are relatively small compared to the signal, leading to very few noise events and a "blind" spot for the algorithm.
  • Hardware Dependency: The trained models are currently bound to the specific bias parameters of the camera used during training. Future work could involve making the model "bias-aware" or using non-parametric models to generalize across different event camera hardwares.
  • Integration Time: To get a good image, you need to aggregate noise for a finite window (the authors used 1 second). While this is fine for static backgrounds, it limits the "framerate" of the static recovery.

Summary updated

Abstract

Event cameras, also known as dynamic vision sensors, are an emerging modality for measuring fast dynamics asynchronously. Event cameras capture changes of log-intensity over time as a stream of 'events' and generally cannot measure intensity itself; hence, they are only used for imaging dynamic scenes. However, fluctuations due to random photon arrival inevitably trigger noise events, even for static scenes. While previous efforts have been focused on filtering out these undesirable noise events to improve signal quality, we find that, in the photon-noise regime, these noise events are correlated with the static scene intensity. We analyze the noise event generation and model its relationship to illuminance. Based on this understanding, we propose a method, called Noise2Image, to leverage the illuminance-dependent noise characteristics to recover the static parts of a scene, which are otherwise invisible to event cameras. We experimentally collect a dataset of noise events on static scenes to train and validate Noise2Image. Our results provide a novel approach for capturing static scenes in event cameras, solely from noise events, without additional hardware.
BibTeX
@misc{cao2024noise,
  author = {Ruiming Cao and Dekel Galor and Amit P. S. Kohli and Jacob L. Yates and Laura Waller},
  title = {{Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras}},
  year = {2024},
  eprint = {2404.01298},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.01298},
}
Figure