Event cameras record a per-pixel signal only when the logarithm of brightness changes, so they see fast motion and extreme lighting that ordinary cameras miss. Most are monochrome. The few color versions, such as the Color-DAVIS346 with a red-green-blue filter mosaic over its pixels, have almost no training data: the main real dataset is 50 minutes at 346×260. The usual fix for scarce event data is to simulate events from ordinary video. That fix carries a hidden flaw for color. Web video is dominated by edges that differ in brightness, and on those edges a monochrome sensor already fires. The cases where color events actually matter are edges that differ in hue but not in brightness. There, a monochrome sensor sees nothing. Such edges are rare in any video corpus, so a network trained on simulated color events can learn to ignore its color channels and still score well.
We propose to generate training video that targets this gap. Before simulating events, we recolor objects in source video while holding each pixel's luminance (its perceived brightness) exactly fixed. The monochrome event stream is therefore unchanged by construction, and only the color-channel events differ. A generator (an image colorizer or diffusion editor) proposes the new colors, but it cannot touch brightness, edges, or motion, so it cannot add hallucinated structure. The question is whether this counterfactual, luminance-preserving augmentation teaches color-event networks to use chromatic information. We also ask whether a learned generator is needed for it, or whether simple procedural recoloring suffices.
Core Claim
Training color-event reconstruction and perception models on simulated data fails in one specific way: on hue-only edges, the data never forces the model to rely on its color channels. Luminance-locked recoloring fixes this cheaply. Let $c_t(x) \in \mathbb{R}^3$ be the linear RGB value of pixel $x$ in frame $t$, and let $w$ be the luminance weights. An edited frame $c'_t$ is admissible only if
$$ w^\top c'_t(x) = w^\top c_t(x) \quad \text{for all } x, t, $$
so luminance is identical while the chroma (the color components left after removing brightness) is free. Under any simulator that thresholds log-intensity changes, the luminance event stream of the edited video equals that of the original. The red, green, and blue Bayer-channel streams differ, and they differ most where the edit creates or removes chromatic contrast. We predict that models trained with such pairs will improve mainly on chromatic edges and luminance-matched objects, not uniformly.
Relation to Prior Work
Two papers supply the simulation machinery. CED collected the real Color-DAVIS346 data and extended the ESIM simulator to model the Bayer mosaic offline. It found that E2VID, a neural network that reconstructs images from events, works best on per-channel quarter-resolution streams. V2V, with code at HYLZ-2019/V2V, converts compressed video into event voxel grids (time-binned event histograms) during training. This gives a 150× storage reduction (about 19 GB instead of about 3 TB for WebVid10k). It also showed that diverse imperfect video beats smaller high-fidelity renders on real benchmarks. Both papers treat the content of the source video as given. V2V is monochrome only.
Porting V2V to Bayer channels is the obvious step at this intersection, and we treat it as our naive baseline rather than our contribution. If "diversity beats fidelity" holds, the remaining limitation is that the diversity lacks the right kind of chromatic content.
The motivating evidence comes from three papers.
- MS-EVS showed that luminance-matched objects produce zero events in monochrome. It also showed that event sensors benefit 2.6× more from extra spectral bands than frame cameras do.
- HUE showed the "perceptual metric trap". Monochrome event reconstructions scored well on NIQE (4.21) and MUSIQ (46.00), two no-reference image-quality scores, yet YOLOv7 hallucinated objects on them.
- Color Event Enhanced Single-Exposure HDR Imaging shows that the color-event field still leans heavily on ESIM-simulated data.
The method of keeping luma fixed and generating only color borrows from two decoupling results.
- ChromaDistill freezes geometry learned from grayscale inputs and distills color from a 2D colorizer. It reports that naive per-view colorization causes color flicker.
- DiET-GS found a "tug-of-war" when diffusion priors were optimized jointly with event-based structure, and had to freeze geometry before generative refinement.
We apply the same lesson to data generation rather than reconstruction: the generator may edit only chroma.
3D MCGAN supplies both the template and a warning. It inserted targeted synthetic lung nodules into real CT scans, raising detection from 0.518 to 0.550 CPM (a detection score averaged over false-positive rates). However, 3× augmentation hurt performance through overfitting to generator artifacts. We therefore sweep the ratio of synthetic to unedited data rather than assuming more is better. Generative event-augmented view synthesis such as EA3D uses monochrome events and static scenes. It addresses a different problem and is not a competitor here.
Research Program
Edit generator. For each WebVid10k clip, we segment and track objects with an off-the-shelf video segmenter. We then produce a recolored clip in one of three ways:
- a procedural method that rotates and rescales chroma inside each tracked mask in a perceptual color space;
- a learned colorizer or diffusion editor that proposes plausible new object colors;
- a targeted "isoluminant" mode that sets an object's luminance to match its local background and assigns it a contrasting hue.
The isoluminant mode is the counterpart of MCGAN's rare-nodule injection. In every case we project the result back onto the constraint above in linear RGB, clamping out-of-gamut pixels and discarding frames where the clamp violates the constraint by more than a set tolerance.
Temporal consistency. This is the main technical difficulty. Frame-to-frame chroma flicker from a per-frame generator (the failure ChromaDistill documents) would itself trigger spurious color events. We therefore generate color only on a keyframe and propagate it along optical flow within the tracked mask. We also measure flicker directly: we count color-channel events in regions where the original video had no luminance change. That count should be near zero after editing.
Simulator. We extend V2V to the RGBG Bayer layout. Each pixel uses only its own filtered channel, with per-channel randomized contrast thresholds and noise. For Gap 5 robustness we add an optional 50/60 Hz flicker injection, following the AC-flicker failure reported in RGB-Event ISP. Edits run before simulation, so the pipeline stays on-the-fly and adds only the cost of segmentation and colorization. Masks and colors can be cached per clip at small storage cost.
Models trained. - A color E2VID reconstructor, in both CED's per-channel variant and a single network taking the full mosaic as input. - A detector run on reconstructions, following CED's YOLO protocol.
Baselines
Naive reference. The naive reference is a Bayer-channel port of V2V trained on unedited WebVid10k with standard flips and crops, at identical compute, number of iterations, and hyperparameter search budget. This is the "do nothing beyond the obvious" point, and every other arm is measured against it.
Existing color-event method. The strongest published method for this setting is CED's recipe: E2VID on per-channel quarter-resolution streams, trained on ESIM-simulated color events CED. We rerun it on the same source clips, subsampled to fit ESIM's storage cost, and report both a size-matched comparison and the comparison at the scale each pipeline can actually afford. No reported result found on Lacuna for numerical CED reconstruction scores. The paper reports only that reconstructions "remain clear" from 0.8 to 100,000 lux and that YOLO distinguishes objects identical in monochrome. We therefore measure this baseline ourselves rather than quote it. For the HDR task, the file reports only margins over prior methods, so no absolute HDR-VDP-3 figure is available to beat.
Real-data fine-tuning. Each pretrained model is also fine-tuned on CED's training split. Beating the baseline means a gain that survives this fine-tuning.
Ablations isolating the key ingredient. - Global hue and saturation jitter. This changes color but creates no new object-versus-background chromatic contrast. - Recoloring without the luminance lock. This lets the generator alter brightness and therefore mono events, which tests whether the lock prevents hallucination damage. - Procedural versus learned recoloring. - A sweep of the edited fraction (0.25, 0.5, 1, and 3 times the unedited data).
What counts as winning. The proposal wins if luminance-locked recoloring beats the naive port and the CED recipe on chromatic-edge color error and on the isoluminant probe set described below, with no loss on standard full-image metrics, and the gain persists after CED fine-tuning.
Why not just…?
Why not just use flips, crops, and CutMix on the existing color event data? Geometric augmentation moves existing chromatic edges around but cannot create new hue-only contrast. MS-EVS makes the physical point that a luminance-matched object produces no monochrome signal at all, so the network must learn the color pathway from examples that contain such objects. This alternative is already the augmentation inside our naive baseline, so its contribution is measured directly. If the probe-set gap closes with it alone, our claim fails.
Why not just use the physics simulator ESIM, as CED and HDR-CE do? The proposal is not a different simulator. Our edits act on the source video and could feed ESIM equally well. The issue is twofold. First, ESIM's offline output does not scale: V2V reports roughly 3 TB for a corpus that compresses to about 19 GB as on-the-fly voxels. Second, more faithful physics applied to the same content still never presents hue-only edges. The ESIM arm, run on edited versus unedited clips at matched size, separates these two factors.
Why not just recolor procedurally and skip the generator? This is a real threat, and we expect it may match the learned generator. Masked chroma rotation under the luminance lock is temporally consistent by construction and cannot hallucinate. If the procedural and learned arms tie, we will report that the lock and the targeted isoluminant mode are the active ingredients and that a generator adds only cost. The learned editor earns its place only if its more plausible, context-dependent colors (natural object hues, varied textures) produce measurably better transfer to real CED scenes.
Why not just reconstruct monochrome and colorize afterward with an off-the-shelf colorizer? HUE shows that sharp monochrome reconstructions mislead detectors, and ChromaDistill shows that per-frame 2D colorization flickers. More fundamentally, a colorizer guesses color from shape, while color events measure it. We include this arm (monochrome V2V-E2VID followed by a pretrained colorizer). It should fail precisely on recolored or isoluminant objects, whose true color is not predictable from shape.
Proposed Experiments
Standard reconstruction. On CED, we train reconstructors in all arms and evaluate against the synchronized color frames from the camera's frame sensor using PSNR, SSIM, and LPIPS (a learned perceptual distance). We add a chromatic error in CIEDE2000 (a standard perceptual color-difference formula), computed separately on pixels near hue edges with small luminance gradient.
Downstream detection. Detection is scored as YOLO agreement between event reconstructions and frames, restricted to objects whose class depends on color.
Isoluminant probe set. The decisive test is a small real set captured on a Color-DAVIS346. Printed isoluminant patterns and painted objects move on a linear slider in daylight, following the EC180 setup in Matching Neuromorphic Events and Color Images via Adversarial Learning. Daylight avoids the flicker that a screen or LED lighting would introduce. Luminance matching is verified with the camera's own frames. If luminance-locked training does not beat both the naive port and the hue-jitter ablation here, the core claim is wrong.
Further transfer. Secondary checks test transfer to two other color-event tasks:
- single-exposure HDR on HDR-CE, with HDR-VDP-3, using the Color Event Enhanced Single-Exposure HDR Imaging architecture with and without our pretraining;
- deblurring on EvDeblur-CDAVIS, against DiET-GS++'s reported MUSIQ of 50.44.
Diagnostics. The augmentation-ratio sweep tests whether MCGAN's saturation effect appears in this setting. The spurious-event count verifies that the edit pipeline itself is not teaching the model to read flicker as color.
References
- CED: Color Event Camera Dataset
- V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation
- MS-EVS: Multispectral event-based vision for deep learning based face detection
- HUE Dataset: High-Resolution Event and Frame Sequences for Low-Light Vision
- Color Event Enhanced Single-Exposure HDR Imaging
- ChromaDistill : Colorizing Monochrome Radiance Fields with Knowledge Distillation
- DiET-GS: Diffusion Prior and Event Stream-Assisted Motion Deblurring 3D Gaussian Splatting
- Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-Based CT Image Augmentation for Object Detection
- EA3D: Event-Augmented 3D Diffusion for Generalizable Novel View Synthesis
- RGB-Event ISP: The Dataset and Benchmark
- Matching Neuromorphic Events and Color Images via Adversarial Learning