Full Color Reconstruction from Monochromatic Event Streams

27 concepts25 papers1 proposal

Traditional digital cameras capture the world in full color but are fundamentally limited by their frame-based nature. When an object moves too quickly, or when the lighting drops too low, they suffer from motion blur and severe noise.

Neuromorphic and specialized high-speed sensors—such as event cameras, spike cameras, and single-photon detectors—offer a radical alternative. Instead of capturing full frames at fixed intervals, an event camera operates asynchronously, with each pixel reporting changes in log-intensity the microsecond they occur. This design yields exceptional temporal resolution and high dynamic range (HDR).

However, these sensors suffer from a major drawback: they are almost entirely colorblind. Because they record only changes in luminance rather than absolute wavelength-specific intensity, they output a sparse, monochromatic stream. Resolving this chromatic information gap is one of the most active frontiers in high-speed computer vision.

The Cost of Color Blindness

The transition from monochromatic to color processing is a long-standing challenge in computer vision. In the mid-1970s, Takayasu Ito recognized that monochromatic digital data lacked the essential chromatic cues required for robust scene understanding and human-like perception in varying lighting conditions (Color Picture Processing By Computer).

In modern neuromorphic sensing, this lack of color information creates severe physical and algorithmic bottlenecks:

  • The Luminance Matching Problem: Because event cameras only trigger events when luminance changes, an object moving against a background of a different color but identical brightness will trigger no events at all. As Saad Himmi et al. point out, this makes objects completely invisible to the sensor.
  • Downstream Perception Failures: While monochrome event streams can reconstruct visually sharp structures, they lack semantic fidelity. Burak Ercan et al. demonstrated that event-based reconstructions often cause downstream object detection models to hallucinate false positives due to the absence of color textures. Similarly, Daniel Gehrig et al. found that the lack of chromatic cues reduces the robustness of long-range feature tracking descriptors.
  • Ambiguity in Environmental Estimation: A monochrome sensor cannot differentiate between a change in light intensity and a change in light color. Consequently, systems designed to estimate indoor lighting cannot determine the color temperature or spectral distribution of light sources (Zehao Chen et al.), making it difficult to match virtual objects to real-world environments.

To overcome these limitations, researchers have developed three primary paradigms: hybrid sensor fusion, algorithmic prior distillation, and hardware-level optical re-engineering.

Hybrid Sensor Fusion

The most intuitive way to restore color to an event stream is to pair the event camera with a standard RGB frame-based camera. In this hybrid setup, the standard camera provides low-frequency color information, while the event camera fills in the high-frequency motion dynamics between frames.

[Standard RGB Camera]  --> Low-Frequency Color Frames \
                                                       --> [Fusion Framework] --> High-Speed HDR Color Video
[Monochrome Event]     --> High-Frequency Dynamics     /

Fusing these two radically different modalities requires accounting for the unique noise profiles of each sensor. Ziwei Wang 0002 et al. addressed this by developing an Asynchronous Kalman Filter (AKF) that models overexposure in standard frames and refractory noise in event streams, successfully reconstructing sharp, full-color HDR video during rapid motion.

When lifting these reconstructions into 3D spaces, such as Neural Radiance Fields (NeRFs), the fusion must be handled carefully. Qi Ma et al. demonstrated that fusing sparse RGB images with continuous event streams allows for the reconstruction of fast-moving, deformable objects. However, they noted that if the high-speed motion data from the event sensor cannot be perfectly aligned with the sparse wavelength-specific intensity changes from the RGB camera, severe color artifacts can emerge.

To bypass the need for heavy deep-learning priors in this fusion process, Gereon Fox et al. framed hybrid reconstruction as a non-learning optimization problem. By using continuous Bézier curves and per-event confidence weights, their method aligns blurry RGB frames with sharp monochrome events to output clean color video without requiring massive pre-training datasets.

Algorithmic Colorization and Prior Distillation

What happens when synchronized RGB frames are unavailable, or when researchers want to reconstruct scenes purely from event data? In these scenarios, frameworks must algorithmically infer color by leveraging structural and semantic priors.

Even advanced reconstruction networks, such as the event-based synthetic aperture imaging system built by Xiang Zhang et al., or the nighttime enhancement models designed by Haoyue Liu et al., are frequently constrained to monochromatic outputs because the underlying physical signals lack color information. Purely physics-based event models, like the self-supervised implicit neural representation framework proposed by Zipeng Wang et al., also struggle to generate color because they must map directly to the underlying event generation equation, which natively lacks color filter array awareness.

To solve this, researchers use knowledge distillation to transfer color knowledge from pre-trained 2D image models into monochrome 3D representations.

For instance, Ankit Dhiman et al. introduced a method that colorizes monochrome radiance fields (such as those from infrared sensors or legacy grayscale captures) by distilling knowledge from powerful 2D image colorization models into 3D structures like NeRFs and 3D Gaussian Splatting. This ensures that the inferred colors remain stable and consistent across different viewing angles.

Similarly, Lin Wang developed a framework that achieves pure event-based 3D Gaussian Splatting. Because event sensors capture log-intensity changes rather than absolute color values, standard optimization leads to desaturated outputs. The author resolved this by distilling structural and chromatic priors from pre-trained event-to-video models, enabling fast, vibrant 3D scene synthesis from purely monochromatic inputs.

Hardware and Optical Re-engineering

Rather than fixing the color deficit in post-processing, a third approach seeks to re-engineer the hardware and optics so that neuromorphic sensors can capture chromatic data natively.

One path is the integration of physical Color Filter Arrays (CFAs), such as Bayer patterns, directly onto the sensor silicon. Jing Zhao et al. explored this in the context of continuous spike cameras, noting that adapting grayscale intensity mapping to handle Bayer-filtered spike streams remains a difficult challenge due to the complex spatial-temporal multiplexing required. A similar hardware limitation affects high-performance single-photon detectors, where the lack of integrated color filter arrays restricts their extreme dynamic range capabilities to monochrome intensity reconstruction (Yuhao Liu et al.).

Alternatively, researchers can expand the sensor's spectral diversity beyond standard RGB. Saad Himmi et al. introduced multispectral event-based vision, capturing data across multiple visible and near-infrared (NIR) bands. They demonstrated that adding spectral diversity provides a much larger performance boost for event cameras than for traditional cameras, effectively preventing objects from disappearing against matching backgrounds.

Monochrome Event Sensor + Standard Lens      --> Captures only luminance changes (Colorblind)
Monochrome Event Sensor + Ball Lens (Aberr.) --> Disperses wavelengths spatially (Spectral Sensor)

Perhaps the most radical optical solution bypasses color filters entirely by mimicking biology. Marine animals like cephalopods are technically color-blind because they have only one type of photoreceptor, yet they interact with a colorful world by exploiting the chromatic aberration of their eyes.

Sami Arja replicated this mechanism by pairing a standard monochromatic event camera with a ball-shaped lens that intentionally induces chromatic aberration. Because different wavelengths of light bend at slightly different angles, high-speed motion causes different colors to hit the monochrome sensor at distinct spatial offsets. By decoding these aberration patterns, the author transformed a standard monochrome event camera into a high-precision spectral sensor without sacrificing the sensor's spatial resolution or light sensitivity.

The Data Bottleneck

Regardless of whether the solution is algorithmic or hardware-based, progress in color event processing is heavily dependent on data availability.

Cedric Scheerlinck et al. provided a foundational resource by releasing the Color Event Camera Dataset (CED), alongside a color-capable event simulator. This allowed researchers to prototype color-aware event algorithms in controlled environments for the first time.

Despite these early contributions, data scarcity remains a major bottleneck. As Zhipeng Tang et al. noted in their work on training neural radiance fields from handheld event cameras, the transition from grayscale to full-color 3D synthesis is severely constrained by the lack of large-scale, high-quality color event datasets. Expanding these datasets and simulation tools remains essential for unlocking the full potential of high-speed, color-aware neuromorphic vision.

Key Foundational Work

Event-Based Color Segmentation With a High Dynamic Range Sensor (2018) — contributed 1 concept elements to this direction.
Color Picture Processing By Computer (1975) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_color_event_dataset_scarcity_fdaf8bd3,
  title = {color event dataset scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/color-event-dataset-scarcity/rdr_fdaf8bd372a34e96b4fe87690989edf2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_color_event_dataset_scarcity_fdaf8bd3_v1,
  title = {color event dataset scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/color-event-dataset-scarcity/rdr_fdaf8bd372a34e96b4fe87690989edf2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake