Reliability-Masked Long-Range Fusion for Event and Frame Cameras

When a car drives at night, some parts of the scene go blind for both sensors at once. A parked car in shadow is too dark for the ordinary camera. It is also invisible to the event camera, a sensor that reports only per-pixel brightness changes, because nothing there is moving relative to the background. Most methods that fuse frames and events combine the two sensors at each location, or within a small neighborhood. Inside such a "joint dead zone" that local combination has nothing to work with. The only usable evidence lies elsewhere: the lit road surface around the car, its moving edge as it enters view, the lane structure. This proposal argues that the receptive field a fusion network actually needs is set by the size of these dead zones, not by object size.

The concrete proposal is a fusion module that does two things. It estimates, from sensor physics, where each modality is trustworthy. It then propagates information over long distances only from trustworthy locations into dead zones, using a cheap row-and-column attention pattern applied at a coarse scale of the feature pyramid. The central test is a new diagnostic: plot per-pixel error against the distance to the nearest reliable pixel. Local fusion should fail as that distance grows. Unmasked global attention should do better. Reliability-masked propagation should keep the curve flat.

Core Claim

Existing frame–event fusion treats sensor failure as something to down-weight. That is enough when at least one sensor works at a given pixel. It does not help when both fail over a region wider than the network's effective receptive field. In that regime, accuracy depends on routing evidence from distant reliable regions, and the routing must be told which sources are reliable. If it is not told, attention keys drawn from saturated glare or flicker-induced event bursts contaminate the fill-in. The claim is falsifiable on three counts. Error at large dead-zone distance should drop substantially relative to local-gating fusion. Masking should beat equal-capacity unmasked attention specifically in that regime. And the gain should vanish on pixels whose nearest reliable neighbor is close.

Relation to Prior Work

Three papers motivate the gap.

Two papers constrain the design.

  • DSEC: A Stereo Event Camera Dataset for Driving Scenarios documents the asymmetric failures: frames degrade at night, while flickering streetlights make event sensors saturate with burst noise. Any event-rate reliability cue must therefore treat too many events as unreliable, not only too few.
  • JL-DCF shows that fusion networks learn "lazily," over-relying on one modality. This is a reason to make reliability explicit rather than hope attention discovers it.

One paper supplies the method.

  • CCNet: Criss-Cross Attention for Semantic Segmentation has each pixel attend only to its own row and column. Two recurrent passes connect every pixel to every other pixel, using 11× less memory and 85% fewer FLOPs than full non-local blocks. CCNet is single-modality and has no masking. We reuse its two-pass routing as the propagation operator and add a reliability bias.

Research Program

Two-stream backbone. The model has a frame stream and an event stream, with events binned into voxel grids as in FRN. Each stream produces a four-level feature pyramid.

Reliability maps. At each pyramid level we compute a per-pixel reliability map for each sensor. The frame map starts from physical cues: distance from intensity saturation and underexposure, and local event density as a blur indicator. The event map is a band-pass function of local event rate. It is low for silent static regions and low for flicker bursts. A small convolutional head refines both maps. To keep the maps tied to physics rather than collapsing into another free attention weight, the head is trained with an auxiliary loss toward the physical proxies.

Masked propagation at the coarsest level. The fused feature at a pixel $i$ is updated by recurrent criss-cross attention with weights

$$ a_{ij} = \mathrm{softmax}_{j \in \mathcal{C}(i)}\left(\frac{q_i^\top k_j}{\sqrt{d}} + \log r_j\right), $$

where:

  • $\mathcal{C}(i)$ is the set of pixels in $i$'s row and column;
  • $q_i$ and $k_j$ are query and key vectors of width $d$;
  • $r_j \in (0,1]$ is the larger of the two sensors' reliabilities at $j$.

Unreliable pixels thus contribute almost nothing as sources, yet still receive information as targets. Running this at 1/16 resolution makes two-pass propagation span the whole image cheaply.

Per-scale gate. On the way back up the pyramid, a gate driven by $1 - r_i$ blends the propagated context with locally fused features. Where sensors are healthy, fine scales keep the sharp edges that events provide. Inside dead zones, propagated coarse context dominates.

Application regime. Semantic segmentation is the primary task, because dense labels let us measure error pixel by pixel as a function of dead-zone distance. Detection is a secondary check.

Baselines

Naive references. The do-nothing reference is the frame-only backbone with the same decoder. We pair it with naive channel concatenation of frames and event voxels, which is the design FRN reports as being swamped by frames. Both are trained with the identical recipe and give the floor.

Strongest published system on the main benchmark. On DSEC-Semantic this is the hybrid ANN–SNN fusion model of Efficient Event-Based Semantic Segmentation via Exploiting Frame-Event Fusion. It processes frames with a standard network and events with a spiking network, and exchanges features through cross-attention injectors. It reports 66.57% mIoU on DSEC-Semantic, 67.31% on DDD17-Seg and 50.91% on M3ED-Semantic. We compare against these reported figures. We also rerun its released configuration, if available, on our dead-zone-stratified evaluation, because a single mIoU number cannot reveal the regime we care about. Beating it means two things: matching or exceeding 66.57% overall, and showing a clearly lower error in the far-from-reliable bins.

Robustness baseline. On DERS-XS we compare to ESC. The file reports only its +2.55 mIoU margin over CMNeXt and no absolute value, so we retrain ESC from its published recipe on our splits.

Reimplemented global cross-attention. We reimplement FRN-style bidirectional global cross-attention on the same pyramid. It gets the same parameter count, training schedule and hyperparameter search budget (an equal number of trials) as our model. FRN's reported 38.0% mAP and 69.5% mPC on DSEC detection serve as the reference for the secondary detection experiment.

Ablations that isolate the key ingredient. Four variants share every other component:

  • the masking term $\log r_j$ removed (plain criss-cross);
  • the reliability maps replaced by randomly permuted maps;
  • propagation run at the finest scale instead of the coarsest;
  • local-only reliability gating with no propagation, which is the ESC-like condition.

Why not just…?

Why not just use full global cross-attention, as FRN does, and let it learn to ignore bad keys? This is the most serious threat. Attention can in principle learn low affinity for saturated or bursting regions. Our bet is that it does so poorly where training signal is scarce, inside large dead zones. The unmasked-criss-cross ablation and the reimplemented FRN fusion settle the question directly. If either matches our model in the far-distance bins, the masking contribution is refuted. We would then report the stratified diagnostic as the main finding.

Why not just enlarge the receptive field with dilated convolutions or a deeper backbone? Guanbin Li et al. argue that deep local stacks dilute long-range signals and discard detail during downsampling. That argument is not specific to dead zones, so we test it here: a dilated-convolution variant at matched FLOPs is included. If its error curve flattens as ours does, long-range routing is unnecessary.

Why not just broadcast a pooled global descriptor, as in Broadcasting Convolutional Network for Visual Relational Reasoning? That method max-pools features to one vector and copies it to every location at linear cost. A single vector cannot say which nearby road surface or edge should fill a particular dead zone. We expect it to help scene-level class priors but not boundaries. A broadcast variant, with reliability-weighted pooling to make it a fair comparison, is included so this expectation is measured rather than assumed.

Why not just fine-tune a large pretrained frame–event model such as CM3AE? CM3AE pretrains a single-scale vision transformer on millions of frame–event pairs. Its pretraining reconstructs frames from masked inputs, which is close to dead-zone filling, so it is a genuine competitor. It lacks a pyramid for dense output, and the file reports no segmentation number for it on DSEC. If compute allows, we fine-tune it with a light pyramid decoder and place it on the same distance curve.

Proposed Experiments

Diagnostic metric. We compute the reliability maps on DSEC-Semantic test frames. For each pixel we measure its distance to the nearest pixel whose reliability exceeds a threshold. Segmentation error is then reported in distance bins, for night and day sequences separately. This curve, not overall mIoU, is the primary outcome.

Controlled dead zones. DERS-XS, the synthetic CARLA set with injected occlusions and degradations, lets us vary dead-zone size directly. We blank both modalities in square regions from 25 to 200 pixels wide and test whether each method's error rises with region size. The prediction is that local fusion and ESC degrade sharply beyond the effective receptive field, while masked propagation degrades gracefully. DERS-XR, the real-world DAVIS-346 set with 240 annotated frames, checks that the synthetic findings transfer. For DSEC-Xtrm there is no reported result in the file, so we would establish our own baselines there.

Decisive checks. Three results decide the outcome:

  • Masked propagation must beat the unmasked ablation in the farthest distance bins on both DSEC night and DERS-XS.
  • The gain must shrink toward zero in the nearest bins. A uniform gain would indicate extra capacity rather than dead-zone filling.
  • Permuted reliability maps must remove the gain.

As a cost check, we report latency and memory at full DSEC resolution against the global cross-attention baseline. CCNet's factorization should keep our overhead modest.