Teaching Event-Camera Networks to Infer Their Own Pixel Thresholds

An event camera does not record frames. Each pixel independently reports a timestamped "event" whenever its log-brightness has risen or fallen by a fixed amount, called the contrast threshold. Simulators usually treat that threshold as one number shared by every pixel. Real chips do not behave this way. Thresholds differ from pixel to pixel, differ between brightening and darkening, and drift as the chip heats up. Networks trained on simulated events therefore meet a sensor they have never seen. The current fixes have real costs. One tunes a single global threshold to match a target dataset's event rate. Another adapts on unlabeled real recordings. A third fine-tunes on a handful of real trajectories. None of them works zero-shot on a new camera.

We propose to treat the threshold map as a hidden, randomized property of the simulated sensor. In every training episode, the simulated camera gets a new, spatially structured, polarity-asymmetric, drifting threshold field. The network is trained to estimate that field from the event stream it receives and to use the estimate in solving its task. This combines domain randomization (training on many randomly varied simulators so the real world looks like one more variation) with a teacher–student scheme in which the hidden sensor parameters are the privileged information. The claim is testable: an optical-flow network trained only on simulation this way should transfer zero-shot across four physically different event sensors. It should do better than randomizing a scalar threshold plus independent per-pixel noise, and its inferred threshold maps should agree with an independent frame-based calibration.

What Is Already Done, and What Is Not

Three lines of prior work motivate the proposal.

The closest is Reducing the Sim-to-Real Gap for Event Cameras. It matches one global threshold to the real event rates of a target dataset and adds background and hot-pixel noise during training. It reports 20–40% better reconstruction on its HQF benchmark. It also documents that the best threshold differs between IJRR and MVSEC. That is direct evidence that a scalar fitted to one dataset cannot cover several sensors, and the method needs target statistics in any case.

v2e simulates per-pixel threshold mismatch. It draws each pixel's threshold independently from a fixed Gaussian, with no spatial correlation, no polarity asymmetry, no drift, and no randomization of the distribution itself.

PECS improves the optics with lens ray tracing and spectral quantum efficiency. It leaves the analog threshold behaviour unmodeled, and it is too slow for large-scale randomized training.

Two other lines constrain the design rather than supply it. Abstraction methods discard the threshold-dependent signal entirely. Binary event masks in Monocular Event-Based Vision for Obstacle Avoidance with a Quadrotor do this, as do ideal ternary events in Beyond Domain Randomization. The first of these still needed real-data fine-tuning. Unsupervised adaptation in DA4Event closes the gap on classification but requires real target recordings.

The direction-level literature also supplies a measurement tool. The Asynchronous Kalman Filter for Hybrid Event Cameras calibrates per-pixel thresholds online using reference frames. We use it as an independent "ruler" for real threshold fields, not as part of our method.

Core Claim

Our hypothesis concerns how a network should respond to unknown, structured sensor variation. Making the network invariant to that variation throws away information. The network should instead infer the variation in context. Concretely, each simulated pixel at location $x$ and time $t$ gets a polarity-specific threshold

$$C_{\pm}(x,t) = C_0 \exp\big(g(x) + a_{\pm}(x) + d(t)\, s(x)\big),$$

where $C_0$ is a global scale, $g$ is a Gaussian random field with standard deviation $\sigma$ and correlation length $\ell$ (the low-frequency gradients across the chip), $a_{\pm}$ is a polarity offset with mean asymmetry ratio $r = C_+/C_-$, $d(t)$ is a slow drift process, and $s(x)$ is a spatial drift profile.

Every episode samples new values of $(C_0, \sigma, \ell, r)$ and new drift settings, on top of randomized leak and latency parameters taken from v2e's pixel model.

A teacher network is given the true field. The student sees only events. It has a small recurrent head that predicts $\hat C_{\pm}(x,t)$, and it is trained to match both the teacher's task output and the true field. The new ingredient is not "more noise". It is spatially structured, explicitly inferred sensor parameters. This directly targets the open problem that current networks absorb threshold variation implicitly in their weights.

Research Program

The program has four stages.

Stage 1: check that the structure exists. On sequences recorded with frame-capable DAVIS sensors (HQF on the DAVIS240C, MVSEC on the DAVIS346), we run the Kalman-filter calibration and fit $\sigma$, $\ell$, $r$ and drift to the recovered threshold maps. This sets the randomization ranges from physics rather than trial and error. It also serves as a kill criterion: if real fields turn out to be nearly independent across pixels and symmetric, the spatial and asymmetric parts of the model are unnecessary, and the proposal collapses to the v2e-randomization ablation below.

Stage 2: build a fast simulator. We render high-frame-rate sequences with ground-truth flow from a rasterized engine. Flightmare reports photorealistic rendering up to 230 Hz, and we will raise temporal resolution by rendering offline. We then run a v2e-style pixel model with the field above. Rendering avoids v2e's frame-interpolation artifacts while staying far cheaper than ray tracing.

Stage 3: train the networks. We train an E-RAFT-style flow network, following E-RAFT, and an E2VID reconstruction network as student models.

Stage 4: closed-loop flight (stretch goal). If the perception results hold, we plug the flow and depth module into the event-based obstacle-avoidance pipeline. The question is whether the real-data fine-tuning that pipeline needed can be dropped.

Baselines

All learned baselines share the same architecture, the same rendered scenes, the same number of synthetic training samples, and the same hyperparameter search budget. Model selection uses held-out simulation plus, at most, one designated validation sensor. The test sensors are never used for selection.

The naive reference point is training on an ideal simulator with one fixed threshold. It shows how large the gap is.

The strongest existing zero-shot-compatible method is the rate-matched global threshold with noise augmentation from Reducing the Sim-to-Real Gap for Event Cameras. We run it in two forms. In the honest form, the threshold is matched to the validation sensor only. In an oracle form, it is matched separately to each test dataset. That paper reports its gains only as relative improvements (20–40% LPIPS on HQF reconstruction, up to 15% on its flow-warp loss). There is no reported absolute result on Lacuna for this method on DSEC or M3ED flow, so we will reproduce it.

The fixed-parameter v2e simulator, with independent Gaussian per-pixel thresholds, is the second physics baseline.

The key ablation is "flat randomization". It randomizes the global threshold over a wide range and also randomizes v2e's independent per-pixel spread, but has no spatial correlation, no asymmetry, no drift, and no inference head. A second ablation keeps the full structured field but removes the inference head. This separates the value of better randomization from the value of in-context estimation.

As abstraction baselines, we train on binary event masks and on ideal ternary events.

As non-zero-shot references, we include DA4Event-style feature alignment given unlabeled target recordings, and E-RAFT trained on real data. For the real-trained model, the file reports a dense end-point error (EPE, the average pixel distance between predicted and true flow vectors) of 0.79 on DSEC-Flow, 0.27 on MVSEC, and 0.934 on the M3ED car_urban_parking sequence (E-RAFT; M3ED).

Beating the strongest baseline means lower zero-shot EPE than both the honest rate-matched baseline and flat randomization on every held-out sensor. We would also count it as strong if the gap to the per-dataset oracle threshold is closed by more than half.

Why not just…?

Why not just add Gaussian jitter or random event drops to ideal simulated events? Signal-independent jitter does not reproduce structured sensor noise. In image sensors, modeling signal-dependent noise was needed for low-light optical flow to generalize (Optical Flow in the Dark). Physics-based event simulators measurably outperform ideal ones: v2e reports a 40% gain in night driving, and PECS reports up to 43% smaller distance to real event distributions than ESIM. We still include a jitter-augmented ideal simulator in the naive arm so the claim is measured here rather than assumed.

Why not just randomize one global threshold over a wide range, plus v2e's independent per-pixel noise? This is the real threat, and it is exactly the flat-randomization ablation. If Stage 1 finds weak spatial correlation and little asymmetry, or if flat randomization matches our method on all four sensors, we will report that structure does not matter. The remaining contribution would then be the measured parameter ranges. Over-broad randomization is also known to dilute learning, so we additionally check whether flat randomization loses accuracy on the validation sensor relative to rate-matched training.

Why not just normalize each pixel's event count by its own running average at test time? If thresholds only rescaled event counts per pixel, this cheap correction would absorb most of the mismatch. It cannot separate a sensitive pixel from a pixel looking at a busy part of the scene, and it ignores polarity asymmetry and drift. We run it as an additional baseline applied to the flat-randomization model. If it closes most of the gap, the inference head is not earning its complexity.

Why not just use binary event masks so thresholds stop mattering? Masks discard event counts and timing. Dense flow relies on rich temporal voxel grids, the binned spatio-temporal histograms of events that E-RAFT builds on. The mask-based quadrotor system still needed real fine-tuning. The mask baseline will show how much accuracy abstraction costs on flow.

Why not just adapt on unlabeled real data? DA4Event reaches 89.24% against a 90.09% supervised upper bound on N-Caltech101 classification. That approach requires target recordings, which are risky to collect for agile flight and unavailable for a new camera. We report it as a non-zero-shot reference rather than a competitor.

Proposed Experiments

The decisive experiment is zero-shot cross-sensor flow. All flow models are trained on simulation alone and evaluated without any adaptation on four sensors: MVSEC (DAVIS346), DSEC-Flow (Prophesee Gen3.1), and M3ED (Prophesee EVKv4), all with ground-truth flow, plus HQF (DAVIS240C), where flow has no ground truth and is scored with the flow-warp loss. Metrics are EPE and angular error where ground truth exists. Reconstruction is evaluated with LPIPS on HQF.

The second check targets the inference head. On HQF and MVSEC, we correlate the student's inferred threshold maps with the Kalman-filter calibration. We also test whether accuracy degrades over long sequences, where drift accumulates, more slowly than for the no-head ablation.

The third check is a stress test. We run the methods on the MVSEC night sequences and the DSEC night sequences, where existing stereo error rises from about 10.5% to 15.47%. We then sweep the simulated asymmetry and correlation length beyond the training range to see where each method breaks.

The proposal succeeds if three things hold: structured randomization with inference beats flat randomization on held-out sensors, the inferred maps correlate with independent calibration, and Stage 1 confirms that real fields are spatially correlated. Failing the third reduces the work to a careful negative result about how much physical detail event-camera simulators actually need.