Event cameras are sensors whose pixels fire only when brightness changes. Networks that estimate optical flow (the per-pixel motion field) from these events are usually trained on simulated events, because simulators give exact motion labels for free. The trained networks then degrade on real hardware. The best-documented reason is that the simulator's "contrast threshold", meaning the brightness change needed to trigger an event, does not match the real sensor's. Real sensors also add background noise and hot pixels that the simulator omits. The standard fix is to tune the simulator to each target camera. Reducing the Sim-to-Real Gap for Event Cameras showed that this tuning works but contradicts itself across cameras: IJRR wanted low thresholds and MVSEC wanted high ones.
We propose to borrow a trick that motion-sensitive neurons appear to use. They compare a motion detector against its mirror image and report the normalized difference. Under that comparison, anything that looks the same in both directions cancels in the numerator, and any overall scaling of the input cancels in the ratio. A threshold mismatch mostly rescales event counts. Background noise and lighting flicker have no preferred direction. Both are therefore precisely the kind of signal this comparison removes. The proposal builds the comparison into the first layer of an event-flow network as a hard weight-sharing constraint. It then tests whether a single synthetic-trained model can transfer to several real cameras without per-camera simulator tuning.
Core Claim
In the first spatiotemporal layer, every learned kernel $k(x,y,t)$ is paired with its spatially mirrored copy $\tilde k(x,y,t)=k(-x,-y,t)$, which responds to the same pattern moving the opposite way. The layer outputs the normalized opponent response
$$ o = \frac{r - \tilde r}{r + \tilde r + \epsilon}, \qquad r = [k * E]_+,\ \ \tilde r = [\tilde k * E]_+, $$
where $E$ is the binned event volume, $*$ is convolution, $[\cdot]_+$ is rectification, and $\epsilon$ is a small constant. Two properties follow directly from the form. First, if the threshold mismatch multiplies event counts by a factor $g$, then $r$ and $\tilde r$ both scale by $g$, so $o$ is unchanged up to $\epsilon$. This rests on the approximation that event count is inversely proportional to threshold for a fixed log-intensity change. Second, spatially uniform flicker, or noise that is statistically symmetric under reflection, adds the same amount to $r$ and $\tilde r$ in expectation. It therefore leaves the numerator unchanged and only inflates the denominator, which acts as a confidence gate.
The claim is that this constraint makes the representation partly invariant to the factors that defeat simulator calibration. As a result, one model trained once on synthetic data should transfer across MVSEC, DSEC and HQF. The benefit should be largest for compact networks, which Reducing the Sim-to-Real Gap for Event Cameras found get worse when trained on wider synthetic distributions.
The idea is not "add biologically inspired direction-selective filters." A free 3D convolution can already learn direction-selective kernels, so that claim would be empty. What is new is the specific tie between a kernel and its mirror, followed by a divisive comparison. That pairing is what produces the invariances above, and it can be ablated on its own.
Where the Pieces Come From
Several papers motivate or supply the mechanism. Direction Opponency, Not Quadrature, Is Key to the 1/4 Cycle Preference for Apparent Motion in the Motion Energy Model shows that the defining behavior of motion-energy models comes from subtracting oppositely tuned subunits, not from carefully phase-shifted filter pairs. That result supplies our design choice: arbitrary learned kernels paired with their mirrors are enough, and quadrature pairs are unnecessary.
An Architectural Mechanism for Direction-tuned Cortical Simple Cells: The Role of Mutual Inhibition and Direction Selective Silicon Retina that Uses Null Inhibition produce direction selectivity through asymmetric inhibition. The first does it in a cortical circuit model, the second in analog hardware. Both motivate inhibition between opponent channels. However, the cortical model's tuning depends on time constants of roughly 10–20 ms, which is a warning that timing-sensitive recurrent circuits may themselves fail to transfer.
Generating velocity tuning by asymmetric recurrent connections is a second constraint. It shows that thresholded asymmetric recurrent fields break into "lurching" waves outside a bounded velocity range. For that reason we use a feedforward opponent stage rather than a recurrent neural field.
Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation is the closest bridge between the two literatures. Its spiking network learns direction-tuned filters from events using multi-delay synapses and runs on both synthetic and real streams. But it produces sparse estimates and global motion classes, and it never measures transfer degradation on MVSEC or DSEC. We produce dense flow and measure the gap directly.
Research Program
We will build two backbones and swap in an opponent first layer. The first is a compact network at FireNet scale. The second is EV-FlowNet, as used by Reducing the Sim-to-Real Gap for Event Cameras. Each opponent layer holds a bank of mirror-tied kernel pairs. Speed is still encoded because different kernels within the bank can prefer different speeds. Static edge information, which the opponent ratio discards, has to reach later layers some other way. Whether it should is itself a question we test: one variant adds a separate non-opponent "sum" channel ($r+\tilde r$), and we measure whether that channel brings back threshold sensitivity.
Training uses simulated events with dense labels, generated by the Stoffregen et al. pipeline in two regimes. In the first, the threshold is fixed at one default with no per-target calibration. In the second, the threshold is randomized over a wide range. All evaluation on real data is zero-shot.
Analysis has three parts. We measure how the first-layer response changes when the threshold is scaled synthetically. We test robustness to uniform flicker events injected into real recordings. We fit the opponent kernels and compare their preferred-speed distribution against the input statistics, and we also report the direction selectivity index used by Sabatini et al.
Baselines
The naive reference is EV-FlowNet and the compact network trained on default-threshold synthetic data with no augmentation. This is the "do nothing" gap that every other method must close.
The strongest sim-to-real competitor is Stoffregen et al.'s calibrated pipeline, which matches the threshold to the target and adds dynamic noise augmentation. It is the only method in the file aimed squarely at this gap, and it reports "up to 15% improvement" for EV-FlowNet measured by Flow Warp Loss. That metric measures how sharp the events become after warping them by the predicted flow, so it needs no ground truth. We run it two ways. In the per-target mode it gets its calibrated threshold for each dataset, which gives it an advantage. In the single-model mode, matching ours, it gets one calibration for all targets. Both arms use identical backbones, synthetic data volume, epochs and hyperparameter search budget.
Beating this baseline means three things together. Our single model must match per-target calibration on its own best dataset. It must beat single-model calibration on the mismatched dataset. And it must not degrade the compact network on IJRR or HQF, where calibration hurt FireNet.
The key ablation is an untied control with the same parameter count. Its first layer has the same kernel count and the same divisive normalization, but the paired kernel is free rather than a mirror copy. This isolates whether the reflection tie matters or whether normalization alone does the work. A second ablation removes the normalization, keeping $r-\tilde r$ only, to separate the noise cancellation from the gain invariance.
For context we report supervised and unsupervised methods trained on real data as reference levels that a zero-shot method should approach, not beat. E-RAFT reports EPE 0.27 on MVSEC and 0.79 on DSEC-Flow (E-RAFT). EPE, or endpoint error, is the average pixel distance between predicted and true flow vectors. The multi-reference focus loss reports AEE 0.42 on indoor_flying1 (Secrets of Event-Based Optical Flow). For TMA's documented outdoor-to-indoor degradation, no reported result was found on Lacuna giving the size of that drop. We will therefore measure it ourselves with the released configuration where feasible.
Why not just…?
Why not just randomize the contrast threshold widely? This is the real threat. A large enough network trained on a broad enough threshold range may learn the same invariance from data. The evidence against it is partial. Stoffregen et al. found that compact networks degrade when given wider distributions, which suggests randomization costs capacity that the tie provides for free. We test this directly by crossing randomization against the opponent layer at both network sizes. If randomization matches the opponent layer at EV-FlowNet scale, we will narrow the claim to compact and low-power models rather than argue around the result.
Why not just normalize event counts per pixel or per window at the input? Input normalization does remove global gain. However, it does nothing about direction-symmetric noise or flicker, and a per-window normalizer is itself disturbed by bursts of flicker. The flicker-injection test separates the two approaches. Input normalization is also included as an extra baseline.
Why not skip simulation and train with contrast maximization on real events? Contrast maximization aligns events along predicted motion to make the warped event image as sharp as possible, so it needs no labels. It removes the gap by construction, and it is competitive: 0.42 AEE on indoor_flying1 from Secrets of Event-Based Optical Flow. But it still needs real recordings from each new sensor. It is also vulnerable to flicker being read as ego-motion, as reported in Unsupervised Event-Based Optical Flow Using Motion Compensation. The two approaches are complementary rather than competing, so we include it as a reference and also test whether the opponent layer reduces its flicker failures.
Why not adversarial alignment of synthetic and real features? The point-cloud evidence in Adversarially Masking Synthetic to Mimic Real is that discriminators learn the "clean versus noisy" shortcut, and a large gap remains: 22.8% versus 55.5% oracle mIoU. Adversarial alignment also needs unlabeled target data for every camera, which defeats the single-model goal. We treat it as lower priority and run it only if budget allows.
Proposed Experiments
All experiments train on synthetic events and evaluate zero-shot. We report EPE on MVSEC indoor_flying1–3 and outdoor_day1, and EPE on DSEC-Flow. On HQF and IJRR, which lack dense flow ground truth, we report Flow Warp Loss.
The decisive check is a cross-target consistency score: each method's worst-case relative degradation across all real targets when it is restricted to one model and one simulator setting. Calibration should lose on this score and the opponent layer should win.
Two controlled stress tests support it. The first rescales real event volumes by subsampling or duplicating events to mimic threshold shifts over roughly a factor of four, and plots EPE against the scale factor. The opponent model's curve should be flat where the baseline's curve rises. The second injects synthetic spatially uniform flicker at 100–120 Hz rates into real sequences. Opponent models should show smaller EPE growth than both the untied control and the input-normalized control.
The idea fails if the untied control matches the mirror-tied model. That would mean normalization, not opponency, does the work. It is also a negative result if wide threshold randomization closes the gap at both network sizes. Either outcome would be reported as such.
References
- Reducing the Sim-to-Real Gap for Event Cameras
- Direction Opponency, Not Quadrature, Is Key to the 1/4 Cycle Preference for Apparent Motion in the Motion Energy Model
- An Architectural Mechanism for Direction-tuned Cortical Simple Cells: The Role of Mutual Inhibition
- Direction Selective Silicon Retina that Uses Null Inhibition
- Generating velocity tuning by asymmetric recurrent connections
- Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception
- E-RAFT: Dense Optical Flow from Event Cameras
- Secrets of Event-Based Optical Flow
- Unsupervised Event-Based Optical Flow Using Motion Compensation
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation