Split-Spectrum Cross-Attention for Deep Multimodal Fusion

When a transformer fuses two sensors, for example camera and LiDAR or video and audio, each cross-attention layer lets tokens from one modality take a weighted average of tokens from the other. Graph learning studies a failure mode of this operation. Repeated neighbor-averaging makes all representations drift toward the same vector, a problem called over-smoothing. Softmax attention does not escape it; it only slows it down. We propose that a familiar multimodal failure is a specific, directional version of over-smoothing: the weaker modality's tokens get pulled into the dominant modality's summary and lose their own detail. This failure shows up as "modality collapse" or the "LiDAR crutch."

The proposal separates two things that standard cross-attention forces to share one weight matrix. The first is the consensus part: what every token of modality A receives as the same summary of modality B. The second is the detail part: the query-specific deviations that let one camera patch attend to a different radar point than its neighbor does. Each part gets its own projection weights, with an explicit constraint on its spectrum. The consensus part is allowed to smooth, and the detail part is constrained to preserve or sharpen high-frequency structure. If the diagnosis is right, deeper fusion stacks should stop trading away sensor-specific detail for agreement. Models should also degrade more gracefully when a sensor drops out.

Core Claim

A cross-attention layer updates modality-A tokens $X_A \in \mathbb{R}^{N_A \times d}$ using attention weights $A_{A\to B} \in \mathbb{R}^{N_A \times N_B}$ (each row sums to one) over modality-B tokens $X_B$, through a single value-output matrix $H = W_V W_O$. We rewrite this update as

$$X_A \leftarrow X_A + \tfrac{1}{N_B}\mathbf{1}\mathbf{1}^\top X_B H_c + \big(A_{A\to B} - \tfrac{1}{N_B}\mathbf{1}\mathbf{1}^\top\big) X_B H_d .$$

Here $\mathbf{1}$ is the all-ones vector. The middle term hands every A token the identical mean of B, which is pure consensus. The last term uses rows that sum to zero, so it acts as a high-pass filter carrying only query-specific information. $H_c$ and $H_d$ are separate matrices.

In a standard layer $H_c = H_d = H$. That tie is where the trouble comes from. Demystifying Oversmoothing in Attention-Based Graph Neural Networks proves that because softmax weights are strictly positive, products of attention matrices contract toward rank one. In our decomposition, this means the detail term decays across depth unless $H$ amplifies it. Setting the Record Straight on Transformer Oversmoothing shows, for unimodal self-attention, that the eigenvalues of $H$ decide whether a layer smooths or sharpens. It also shows that a controlled eigendecomposition reparameterization can steer this.

We apply that reparameterization to $H_c$ and $H_d$ separately, inside rectangular cross-attention. $H_c$ is permitted smoothing eigenvalues and $H_d$ is constrained to sharpening ones. What is new is the claim that cross-modal agreement and within-modality distinctiveness need opposite spectral behavior. A single $H$ cannot deliver both, and untying them lets fusion go deeper without the weak modality being absorbed.

Prior Work and How This Differs

Three graph papers motivate the high-pass idea but work on square, fixed graphs within one modality:

Our split is the bipartite, learned-attention analogue of Filterbank GNNs. Unlike PairNorm, it does not rescale features after the fact. It changes which part of the cross-modal message each weight matrix can amplify.

The transformer over-smoothing paper by Dovonon et al. supplies the reparameterization we reuse. It treats only self-attention and ties smoothing and sharpening to one $H$. Its CIFAR100 result, where a forced-smoothing 48-layer model gained +2.62% versus +1.81% for forced sharpening, is a warning we take seriously: smoothing is not simply bad. Our design keeps smoothing where it plausibly helps, in the consensus term, rather than suppressing it everywhere.

Two multimodal collapse papers constrain the design. A Closer Look at Multimodal Representation Collapse attributes modality collapse to polysemantic shared neurons and rank bottlenecks. It fixes these at the encoder level with an adversarial modality discriminator. Countering Multi-modal Representation Collapse through Rank-targeted Fusion measures collapse with effective rank, the exponentiated entropy of normalized singular values. Its fix blends low-information channels across RGB and depth in ResNet features. Neither examines token dynamics across the depth of a transformer fusion stack, which is where our mechanism acts.

Attention Bottlenecks for Multimodal Fusion (MBT) routes cross-modal exchange through a few latent tokens starting mid-network. In our terms, it limits how many layers of cross-modal averaging occur. It is therefore both a baseline and a competing explanation. Finally, Are we measuring oversmoothing in graph neural networks correctly? warns that cosine similarity and Dirichlet energy conflate semantic agreement with rank collapse. For that reason our measurements are rank-based and computed separately per modality.

Research Program

The first stage is diagnostic and could falsify the premise before any architecture is built. We will train standard deep cross-attention fusion models at fusion depths of 2, 4, 8, and 12 layers on audio-visual and camera-LiDAR data, and track three quantities layer by layer:

  • Per-modality effective rank: $\mathrm{erank}(X_A)$ and $\mathrm{erank}(X_B)$, computed separately for each modality.
  • Absorption: the fraction of the weak modality's token energy lying in the span of the dominant modality's top singular vectors.
  • Detail-term norm: the norm of the detail term relative to the consensus term.

The premise predicts two things. The weak modality's effective rank should fall and its absorption should rise with depth. Absorption measured on clean data should also predict the accuracy drop when the dominant sensor is removed at test time. If neither holds, the proposal becomes a negative result about the graph analogy.

The second stage builds split-spectrum cross-attention as a drop-in replacement for cross-attention blocks in two backbones. The first is an MBT-style audio-visual transformer, used in both bottleneck and full-attention variants. The second is the CMT decoder from Cross Modal Transformer. $H_c$ and $H_d$ are parameterized through the Dovonon et al. eigendecomposition, with sign constraints on their eigenvalues.

We will run the modalities under both shared Pre-LayerNorm and per-modality LayerNorm. Dovonon et al. show that normalization weights can flip sharpening into smoothing, and sensors with very different token statistics make that risk larger. The added parameter cost is one extra $d \times d$ matrix per cross-attention block. The consensus term costs $O(N_B d)$ because it is a mean.

Baselines

The naive reference is a full cross-attention fusion transformer with the same backbone and fusion depth and no intervention. Every gain is measured against it at each depth.

The strongest audio-visual competitor is MBT itself. It reports 64.1% top-1 on VGGSound, 43.4% action accuracy on Epic-Kitchens-100, and 52.1 mAP on AudioSet (Attention Bottlenecks for Multimodal Fusion). It is the strongest competitor because its bottleneck and mid-fusion design already limits cross-modal averaging. For driving, the strongest reference is CMT. It reports 74.1% NDS and 72.0% mAP on the nuScenes test set with its VoVNet variant, 72.9% NDS on validation, and 44.7% NDS under simulated LiDAR failure (Cross Modal Transformer). The last number is the one our mechanism most directly targets.

We will reimplement MBT and CMT at reduced scale with the same pretrained encoders, training schedule, and data. Each method, including ours, gets the same number of hyperparameter trials. For MBT this search covers bottleneck count and fusion onset layer; for ours it covers spectral constraint strength. Published numbers are sanity checks, and claims rest on our matched reruns.

The collapse-specific baselines are RTF-style rank-targeted channel blending and EBR-style adversarial basis reallocation, both applied to the same backbone. EBR reports a ~3% AUC-ROC gain on MIMIC-IV with missing modalities, and we find no absolute figure on Lacuna for its avMNIST results. The key ablations are:

  • Untied, unconstrained: $H_c$ and $H_d$ untied but unconstrained, which separates the value of untying from the value of the spectral constraint.
  • Tied, constrained: a single $H$ with Dovonon-style sharpening, which is the direct unimodal method transplanted.
  • Reversed constraints: consensus sharpened and detail smoothed, a sign-flip that tests whether direction matters.

Beating MBT means a gain in VGGSound top-1 or Epic-Kitchens action accuracy that exceeds seed variance at matched compute. Beating CMT means a clear gain in NDS under LiDAR dropout, above the matched 44.7%-equivalent rerun, without losing more than a few tenths of NDS in full-sensor mode.

Why not just…?

Why not just use fewer fusion layers? This is the most serious threat. If collapse is driven by depth, MBT-style mid-fusion with a shallow cross-modal stack may already capture most of the benefit. We settle it by comparing against the best shallow-fusion configuration found by search, not a fixed one. The proposal only earns its keep if split-spectrum models either beat the best depth for the baseline, or turn added fusion depth into accuracy where the baseline's curve flattens or declines. If shallow fusion matches us everywhere, we will report the diagnostic and drop the architectural claim.

Why not just train with modality dropout? Randomly masking sensors during training is the standard fix for crutch dependencies, and CMT already uses masking to reach its failure-mode number. Dropout changes what the model sees; it does not stop the detail term from decaying across layers when both sensors are present. We will combine both methods. If dropout alone matches ours on clean and dropped-sensor accuracy and also removes the absorption signature from stage one, the mechanism is redundant. We consider this plausible for dropped-sensor robustness and less likely for clean accuracy at depth.

Why not just apply PairNorm or per-modality LayerNorm? Both prevent total collapse of token spread. PairNorm restores pairwise distance uniformly, however, so it cannot tell cross-modal consensus from lost detail. Per-modality LayerNorm is included directly as a condition and as a baseline. If it alone closes the gap, the Pre-LN inversion identified by Dovonon et al. is the real culprit, which would itself be a useful finding.

Why not just use RTF or EBR? Both fix collapse at the encoder or channel level, before or outside the fusion stack. They are run as matched baselines. If they remove within-fusion absorption as measured in stage one, our layer-level account is unnecessary.

Proposed Experiments

The primary testbed is VGGSound and Epic-Kitchens-100 with the MBT-style backbone. We report accuracy against fusion depth, per-modality effective rank curves, and accuracy with audio or video removed at test time.

The second testbed is nuScenes with a reduced CMT, reporting NDS and mAP in full mode and under LiDAR failure. The third is KAIST pedestrian and DroneVehicle RGB-infrared detection with a C²Former-style cross-attention block, a setting where infrared carries decisive detail at night. There C²Former reports 28.39 miss rate (All) on KAIST and 74.2% mAP on DroneVehicle (C²Former). avMNIST serves as a cheap setting for full depth sweeps and seed variance.

Three checks decide the question:

  • Absorption as a predictor. Absorption in the weak modality must track sensor-removal degradation across baseline models.
  • Constraint versus untying. The constrained split must outperform the untied-unconstrained ablation, which shows that the constraint rather than extra parameters does the work.
  • Direction of the constraint. The reversed-constraint model must do worse, which tests the claim that consensus should smooth and detail should sharpen, the multimodal version of the Dovonon et al. smoothing-helps observation.