When two cameras sit a few centimetres apart, such as an ordinary RGB camera next to an event camera (a sensor that reports per-pixel brightness changes at microsecond resolution), the same object appears at different pixel positions in the two images. The shift grows as objects get closer to the rig. Most fusion layers combine the two images with small convolutions or with offsets predicted by small convolutions. These layers can only find the matching feature if it lies within their receptive field, meaning the patch of the image a unit can actually see. Once the shift is larger than that patch, the layer is fusing the object in one image with background in the other. The usual fixes go to two extremes. Full cross-attention lets every pixel look everywhere, which costs time quadratic in pixel count. Rectified epipolar attention looks only along image rows, which is cheap but correct only if the cameras were perfectly calibrated and aligned.
This proposal targets the gap between those extremes. For a rigid rig, geometry already says where a match can lie: on a single line in the other image (the epipolar line), whose direction and position follow from the cameras' relative pose. This holds whether or not the images are rectified. We propose to learn that line geometry from data as a handful of global parameters, refreshed per sequence to absorb vibration and drift. Each pixel then attends only to a thin band of the other modality along its own predicted line. The result is a receptive field matched to the misalignment rather than to the image: long enough to cover any parallax, narrow enough to cost roughly linear time. Because the band is computed at full resolution, it also does not lose small targets the way coarse-to-fine search does.
Core Claim
Our claim is that fusion error under sensor misalignment is governed by one ratio: the true cross-modal displacement divided by the receptive field of whatever module discovers correspondence. Local fusion, deformable fusion and axis-aligned factorized attention all fail once this ratio exceeds about one. A band whose orientation comes from learned rig geometry keeps the ratio below one at a cost of $O(N L w)$ rather than $O(N^2)$. Here $N$ is the number of feature-map positions, $L$ the band length, and $w$ its width in pixels.
Concretely, for an RGB feature at position $p$, the candidate event features are
$$B(p) = \{\, q \;:\; d(q, \ell_\theta(p)) \le w,\; \|q - \hat p\| \le L \,\},$$
where $\ell_\theta(p)$ is the epipolar line of $p$ under a learned fundamental matrix with parameters $\theta$, and $d$ is point-to-line distance. The point $\hat p$ is $p$ mapped through a learned global homography, which handles the depth-independent part of the offset. Attention runs only over $B(p)$. A small cross-resolution resampling step handles the different sensor sizes, for example $1440\times1080$ RGB against $1280\times800$ events in CRSOT. We supervise $\theta$ only through the task loss, plus an optional consistency term on band-attention peaks; we use no contrastive alignment loss. That choice follows Negate or Embrace. That paper models misalignment as a latent-variable problem and proves that contrastive objectives discard modality-specific variables, which are exactly the event and thermal cues fusion is meant to keep.
Relation to Prior Work
Three papers motivate the gap. CRSOT builds the only real physically unaligned RGB-event tracking benchmark. Its tracker uses full ViT attention and handles misalignment passively: it inflates predicted feature variance in regions that disagree. The authors name learnable alignment modules as future work, and the tracker reports a 470MB footprint at 32 FPS. STTrack models multimodal tracking with Mamba state propagation and explicitly lists robustness to sensor misalignment as open; its background-suppression module assumes registered grids. Coherent Event Guided Low-Light Video Enhancement does correct baseline offsets, but through an all-pair 4D correlation volume and flow warping. That is the expensive extreme, and it degrades when events become sparse.
Three papers supply the mechanism and tell us what not to copy. ECSIC restricts stereo cross-attention to rows and states that it depends on rectification. We keep its 1D search but make the line learned and oblique. CCNet shows that row-plus-column attention recovers global context in single-modality segmentation at $O(N\sqrt N)$. Its axes, however, are fixed to the image grid, so under an oblique baseline they cut across the true correspondence direction. DAS gets linear-cost global-ish sampling by predicting deformable offsets from local features. That is precisely the design our ratio argument predicts will fail for large offsets, because the offset predictor itself has a local receptive field.
Two papers constrain the design. PWRF shows that horizontally and vertically factorized capsule routing survives synthetic "LiDAR jitter" on DELIVER. That success under small jitter tells us the interesting regime is large, depth-dependent parallax, not jitter. Butterworth as Attention gives global receptive fields through spectral gating but assumes registered bands. Its argument that ViTs act as low-pass filters is one reason we keep full-resolution band features instead of pooled tokens.
Research Program
We will build a band cross-attention fusion block, a per-sequence geometry head, and a misalignment-to-receptive-field diagnostic.
The fusion block drops into an existing RGB-event tracker. We will use a CRSOT-style ViT backbone so that only the fusion operator changes.
The geometry head is a tiny network. It reads globally pooled, low-resolution correlations between the two modalities and outputs the homography and fundamental-matrix corrections. A slow running average then lets vibration and thermal drift be tracked within a sequence.
The diagnostic estimates true displacement per target, from annotated box centres in both modalities where available, or from depth on DELIVER. It also estimates the effective receptive field of each baseline's correspondence module from input gradients. We then plot accuracy against their ratio.
Temporal asynchrony enters as a second axis. A clock offset $\Delta t$ moves a target by roughly its velocity times $\Delta t$, a displacement that is not epipolar. We therefore give the band a learned 2D width that widens with estimated target speed, and we test explicitly whether this suffices or whether temporal offsets need a separate mechanism.
Baselines
The naive reference point is pixel-wise concatenation fusion inside the same backbone with no alignment. VisEvent reports that this kind of fusion barely helps even on synchronized data: 0.627 PR / 0.426 SR, against 0.605 / 0.412 for RGB-only MDNet.
The strongest existing method is the CRSOT tracker, which reports 74.2% PR and 61.8% SR at 32 FPS and 470MB on the CRSOT dataset (CRSOT). We will retrain it with our data splits and schedule, and report its released numbers alongside ours. Beating it means matching or exceeding 61.8% SR at clearly lower memory and higher FPS, or exceeding it at equal cost. Most importantly, it means a significantly better SR on the large-parallax subset of sequences, where we predict the gain concentrates.
We will also run a full dense cross-attention fusion block in the same backbone. This is the receptive-field upper bound and isolates cost from accuracy.
Three structured baselines complete the set. The first is a learned global homography followed by local fusion, which tests whether a depth-independent alignment is enough. The second is coarse-to-fine deformable cross-attention, with offsets refined over a three-level pyramid. The third is a CCNet-style criss-cross cross-modal block.
Two ablations isolate the key ingredient. One fixes the band orientation to image rows, an ECSIC-like setting. The other replaces the learned line with a square window of equal area.
All variants share the backbone, pretraining, training data, epoch count, and a matched hyperparameter search of the same number of trials. FPS and peak memory are measured on the same GPU.
Why not just…?
Why not just calibrate offline and apply a homography? A homography is exact only for a single scene plane, so its residual grows with depth range and baseline. MODISSA and CRSOT both describe physically separated rigs with vibration and drift. This alternative is a real threat if CRSOT scenes are mostly distant. We will measure the residual displacement after the best per-sequence homography on CRSOT. If that residual is consistently below the receptive field of plain local fusion, the band adds nothing, and we will report it as a negative result and pivot to the temporal-offset axis.
Why not just use full cross-attention, as CRSOT effectively does? It solves the reach problem, and we include it as the accuracy ceiling. The objection is cost and resolution. CRSOT reports 470MB and 32 FPS, and VisEvent's transformer fusion on RT-MDNet runs at about 14 FPS. Our claim is only that band attention reaches the same accuracy at a fraction of the cost. If dense attention is clearly more accurate at equal input resolution, the line constraint is losing real correspondences and the claim fails.
Why not just use coarse-to-fine deformable alignment, like flow pyramids? This is the most serious competitor, because pyramids also escape the local receptive field. The specific prediction separating it from bands is about small targets. A small target, or thin event structure, that vanishes at the coarse level gives the pyramid nothing to lock onto, while a full-resolution band still contains it. Low light makes this worse: Liang et al. note that flow-style correspondence collapses once signal is sparse. We will stratify results by target size. If the pyramid matches bands on small targets too, we concede that the geometric prior buys efficiency only, not accuracy.
Proposed Experiments
The first decisive check is the ratio curve. We will build a controlled suite on DELIVER, which provides depth. We warp the event modality with true depth-dependent parallax from virtual baselines at several lengths and orientations, including oblique ones, and add clock offsets that shift the event window by fixed numbers of frames. For every method we plot mIoU against the displacement-to-receptive-field ratio, anchored to PWRF's reported 66.47% clean and 66.2% under LiDAR jitter (PWRF). The hypothesis is confirmed if local, deformable and criss-cross fusion degrade sharply near ratio one while band fusion stays near its clean score. It is refuted if the degradation is gradual and shared by all methods.
The second check is CRSOT itself, reporting PR, SR, FPS and memory against the 74.2 / 61.8 / 32 FPS / 470MB reference, broken down by estimated parallax and target size.
The third check confirms that nothing is lost when sensors are already aligned. On VisEvent the band should match aligned fusion, with STTrack's reported 61.9% AUC (STTrack) as context. We will also report COESOT, where MDNet+CMT reports 0.665 / 0.533 PR/SR. On VisEvent we additionally inject synthetic temporal offsets to see whether the speed-adaptive band width handles asynchrony or whether it must be treated separately.
Finally, the geometry head is checked directly. Its learned epipolar lines should pass within a few pixels of annotated cross-modal box centres on CRSOT. If they do not, the band is succeeding for reasons other than geometry, and that has to be explained before any claim is made.
References
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning
- CRSOT: Cross-Resolution Object Tracking Using Unaligned Frame and Event Cameras
- Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking
- Coherent Event Guided Low-Light Video Enhancement
- ECSIC: Epipolar Cross Attention for Stereo Image Compression
- CCNet: Criss-Cross Attention for Semantic Segmentation
- DAS: A Deformable Attention to Capture Salient Information in CNNs
- Part-Whole Relational Fusion Towards Multi-Modal Scene Understanding
- Butterworth as Attention: Anisotropic Spectral Gating for Pansharpening
- VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows
- MODISSA: a multipurpose platform for the prototypical realization of vehicle-related applications using optical sensors