Teaching Frame–Event Cross-Attention Where Not to Look, Using Synthetic Occluders as Free Labels

Many systems combine two sensors, for example a normal camera and an event camera (a sensor that reports per-pixel brightness changes asynchronously instead of full frames). They often fuse the two streams with cross-attention, where features from one sensor act as queries that pull information from the other sensor's features. Real occlusions damage this fusion in a structured way. A hand passing in front of a tracked object blocks camera pixels and creates a burst of edge events from the hand itself. A lens flare saturates the camera but leaves events intact. A dead sensor region silences events while the camera still sees. In each case the right behavior is specific: ignore the damaged keys, and retrieve the missing content from the stream that still observes it. Natural training data rarely contains these cases, and when it does, nothing labels which keys are damaged.

We propose to generate the occlusions ourselves, consistently across both sensors, and to treat the known occluder geometry as supervision for the attention maps. A synthetic occluder is rendered into the RGB frames and simultaneously into the event stream, following the physically grounded event synthesis of ShapeAug. Because we placed it, we know exactly which keys in each modality are corrupted. We then add a loss that penalizes cross-attention mass on those keys. The core claim is that this explicit mask-derived attention supervision gives occlusion robustness that augmentation alone, gating alone, and attention-guided erasing do not. It should help most on occluders whose shapes and motions were never seen in training.

Where this sits

The existing work at this boundary uses occlusion and attention in the opposite direction, or uses only one of them. Improving Referring Expression Grounding With Cross-Modal Attention-Guided Erasing reads the cross-attention map to decide which word or image region to erase. Attention picks the occlusion. We invert the arrow: a known occlusion supervises the attention. That paper also works on pre-extracted 2D box features and discrete words, not raw sensor streams. CorrBEV drops pixels at random to simulate occlusion and aligns the damaged features with language prototypes. Its occlusions are unstructured and apply only to the camera, and it never computes attention between two physical sensors. PointAugmenting is the closest precedent for keeping an augmentation geometrically consistent across sensors: its pasted objects are occluded by depth order in both LiDAR and image. But it fuses by concatenation, so there is no attention to supervise. ShapeAug supplies our event-side occlusion engine, but it is strictly unimodal and uses linear trajectories with uniform-intensity shapes. BEVFusion (Liang et al.) uses sensor failures only as test-time corruptions, not as a training signal.

The application regime comes from Object Tracking by Jointly Exploiting Frame and Event Domain. Its Cross-Domain Feature Integrator (CDFI) uses bidirectional cross-attention blocks between frames and event voxels, plus an illumination-based weighting. It trains only on natural degradations. The research file names two failure modes we target. When keys from one stream are zeroed or noisy, query–key products disperse attention and spread noise through the network. Separately, Gated Cross-Attention shows that softmax cross-attention between mismatched inputs can wash out the original features. That paper supplies our strongest architectural competitor. Understanding Masked Image Modeling via Learning Occlusion Invariant Feature motivates the view that masking itself drives invariance. It also warns, through its linear-probe gap, that training-only artifacts which are absent at test time can hurt. For that reason our method adds no mask tokens and nothing at inference.

Core Claim

Cross-attention fusion fails under occlusion because nothing tells it which keys are invalid. Synthetic occluders that are physically consistent across both sensors provide that information for free.

The simulator produces three occlusion types per clip. In a shared occlusion, an opaque textured body blocks both sensors and its edges generate events. In a camera-only occlusion, a saturating flare or blur patch hits the frames while events stay intact. In an event-only occlusion, a dead region suppresses events while frames stay intact. Let $a_{ij}$ be the cross-attention weight from query token $i$ in one modality to key token $j$ in the other. Let $c_j \in [0,1]$ be the fraction of key $j$'s receptive area that the simulator marks as corrupted, or as belonging to the occluder rather than the scene. We add

$$ \mathcal{L}_{\text{att}} = \frac{1}{|Q|}\sum_{i \in Q} \sum_{j} a_{ij}\, c_j , $$

averaged over all cross-attention blocks and both directions. Here $Q$ is the set of query tokens. This loss pushes attention mass away from keys we know are invalid. In a camera-only or event-only occlusion, the softmax redistributes that mass toward the intact stream at the same location. The tracking loss is unchanged, and inference uses the unmodified network.

Research Program

We will build the cross-modal occluder simulator on ShapeAug's event synthesis. ShapeAug removes background events inside the occluder, generates polarity events from intensity differences along the occluder's edges, and adds random event dropping. We extend it in two ways. First, occluders get textures and non-linear, spline-based trajectories, which addresses ShapeAug's stated limitation. Second, the same occluder is alpha-composited into the aligned RGB frames.

We then train CDFI with four variants: augmentation only, augmentation plus $\mathcal{L}_{\text{att}}$, augmentation plus gated cross-attention, and all three together.

The analysis tracks attention mass on corrupted keys over training. This answers whether augmentation alone already learns to look away. If it does, the loss is unnecessary and we will say so. A secondary transfer study applies the same recipe to RGB-depth grasping with the local cross-attention of RGB-D Grasp Detection via Depth Guided Learning with Cross-modal Attention. There, depth holes play the role of event-only corruption. This tests whether the idea is specific to frame–event tracking.

Baselines

The naive reference point is CDFI trained exactly as published, with no synthetic occlusion. We retrain it in our pipeline on FE108 so that schedule and seeds match. We found no reported result on Lacuna for CDFI's success rate or precision on FE108, so we will report our reproduced number and compare everything against it, not against a quoted figure.

The strongest augmentation competitor is independent per-modality augmentation. It applies Random Erasing (Random Erasing Data Augmentation) to frames and ShapeAug to events, each drawn independently, with the same total occlusion budget as our simulator. This is what a practitioner would do by default. Beating it shows that cross-modal consistency matters.

The strongest architectural competitor is CDFI with its cross-attention blocks replaced by gated cross-attention, which keeps a learnable fraction of the original features and resists wash-out. We run it with and without our simulator.

The strongest adaptive-erasing competitor is a port of CM-Att-Erase: during training, erase the most-attended region in each modality. This method also uses attention and erasure together, so it is the closest prior idea.

The ablation that isolates our key ingredient is the simulator without $\mathcal{L}_{\text{att}}$. A second ablation uses $\mathcal{L}_{\text{att}}$ with only shared occlusions, to test whether the one-sided cases are what teach cross-modal retrieval.

All variants get the same epochs, the same backbone, and the same hyperparameter search of twelve configurations over augmentation probability and loss weight, selected on a held-out validation split. Beating a baseline means a gain in success rate on the held-out occluder test that exceeds three-seed variation, with no drop in clean FE108 accuracy beyond that variation.

Why not just…?

Why not just use modality dropout, zeroing a whole stream at random? Whole-stream dropout teaches the network to work from one sensor alone. It never creates the case that matters most: a sensor that is valid in some places and corrupted in others, where the network must attend selectively. Our event-only and camera-only occlusions are the partial versions of dropout. We include whole-stream dropout as an extra baseline. If it matches our method on held-out occluders, the spatial structure of the occlusion is not doing the work.

Why not just augment each modality independently with existing tools? This is the most direct threat. If cross-sensor consistency adds nothing, independent Random Erasing plus ShapeAug will match our augmentation-only variant. The independent-augmentation baseline under the same budget settles this. If it wins, we still test whether $\mathcal{L}_{\text{att}}$ helps on top of it, since the loss needs only known corruption masks, not consistency.

Why not just use gated cross-attention and skip the augmentation? Gating lets a query fall back on its own features when the context is poor. That handles wash-out, but it does not tell the query which keys are bad. A gate mixes in the attended context as a whole, so if corrupted keys dominate that context the gate can only reject everything. We expect gating and the attention loss to be complementary, and the factorial design shows whether they are. If gating alone closes most of the gap, the honest conclusion is that architecture suffices, and we will report it.

Why not just use attention-guided erasing like CM-Att-Erase? That method creates hard examples but gives no target for where attention should go afterward. The research file also notes that erasing the single dominant cue can destroy the only informative signal. Our occluders follow geometry, not the model's own attention, so they cannot always remove the one discriminative cue. The direct comparison on held-out occluders decides between the two.

Proposed Experiments

The primary benchmark is FE108 frame–event tracking, measured by success rate and precision. We use three test conditions. The first is clean FE108. The second is FE108 with occluders from the training families, applied at fixed seeds. The third is FE108 with held-out families: articulated or deformable silhouettes, curved Bézier-shaped occluders in the spirit of the polygonal occlusions surveyed in the file, and trajectories with acceleration. Every variant is reported in every condition. The secondary benchmark is GraspNet-Planar, reporting grasp average precision under real and synthetically enlarged depth holes.

Three checks are decisive. First, the attention-loss variant must beat augmentation-only on held-out occluders while staying flat on clean data. Second, measured attention mass on corrupted keys must fall faster with the loss than without, and the size of that drop should predict the per-clip tracking gain. If attention moves but tracking does not, the mechanism story is wrong. Third, removing the one-sided occlusion types should cost more than removing shared ones. If not, the claim that the loss teaches cross-modal retrieval, rather than generic robustness, fails. We would then narrow it to a simpler statement about suppressing occluder distractors.