Models that combine two data streams, such as the sound and the frames of a video, often use cross-modal attention, a layer in which features from one stream decide how much to read from each part of the other. These models are known to lean lazily on whichever stream is easiest, and to throw away whatever they decide is background. A standard fix for "the network discarded information it needed" is reconstruction-based regularization: attach a small decoder that must rebuild the inputs from the internal representation, so the representation cannot become degenerate. Adding that decoder to a cross-attention model brings a second problem. The reconstruction loss and the task loss pull in different directions, and the balance between them, a weight usually set by hand, tends to need re-tuning every time the image or audio encoder is swapped.
We propose that this conflict is not spread across the whole network. It is concentrated in the fusion module, and specifically in the attention weights. The task gradient wants attention to suppress background segments. The reconstruction gradient wants attention to stay spread out so that everything can be rebuilt. If this is true, the fix is narrow. Resolve the conflict only inside the fusion parameters, and let the two gradients add normally everywhere else. A model trained this way should keep the robustness benefits of reconstruction, should be much less sensitive to the loss weight, and should carry over across convolutional and transformer encoders without re-tuning. The same hypothesis also gives a concrete explanation for a published negative result about gradient surgery on multimodal models with separate encoders.
Core Claim
Our central claim is a testable localization hypothesis with a method attached. In a cross-attention fusion model trained with a classification loss plus a decoder that reconstructs both modalities' features from the fused representation, gradient conflict should have three properties. The angle between the two losses' gradients should be negative mainly on the cross-attention and gating parameters. It should be close to zero or positive on the modality-specific encoders. And this pattern should hold for both convolutional and transformer encoders. Applying conflict resolution only to that module should be at least as good as applying it globally. It should be clearly better than a hand-tuned weight once results are averaged over encoder choices rather than tuned per encoder.
The claim builds on an existing observation. Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization projects conflicting gradients of pairwise contrastive losses onto each other's normal plane, following PCGrad. The method helps on a single shared transformer, VATT, but hurts cross-modal retrieval when video, audio and text have separate encoders. Our reading is that global surgery on separate encoders mixes gradients that do not really conflict with the few shared parameters that do, so the projection removes useful signal. This motivates the proposal. That paper also uses no reconstruction loss, and that limits it: its finding may not transfer to a decoder-versus-task conflict, and we test this directly.
Research Program
The base model follows the audio-visual event localization line on the AVE dataset, which labels each second of 10-second YouTube clips with one of 28 events that are both audible and visible. Audio-Visual Event Localization in Unconstrained Videos introduced the dataset with audio-guided visual attention. Cross-Modal Attention Network for Temporal Inconsistent Audio-Visual Event Localization added a "modality sentinel" that lets the model ignore a conflicting stream. Cross-modal Background Suppression for Audio-Visual Event Localization (CMBS) gates out background at the region, time and event level. These papers supply the fusion architecture, and we reimplement CMBS-style gated cross-attention as the shared fusion module. None of them reconstruct anything. That gap is our opening: once a segment is suppressed, its contextual information is gone, and nothing pushes back if suppression becomes over-eager or one modality is starved.
On top of the fused per-second representation we attach a light decoder that reconstructs both the audio and visual encoder features, not raw pixels or spectrograms. We choose features because the reconstruction-metric mismatch noted in the reconstruction literature makes pixel losses favor blur, and feature targets keep decoder cost small. The decoder idea comes from Domain Separation Networks, which rebuilds inputs from shared and private codes to stop information from being thrown away. Disentangled Representation Learning for Environment-agnostic Speaker Recognition shows that a reconstruction anchor stabilizes adversarial training. The closest multimodal precedents are LASPA, which regularizes prefix-tuned cross-attention between speaker and language streams with spectrogram reconstruction, and MDNet, which reconstructs physiological signals after attention fusion. Both combine four or more losses with fixed weights. Neither measures where the losses conflict or tests whether the weights survive an encoder swap. Domain Separation Networks explicitly reports that its loss balance is fragile and dataset-specific. That fragility is what we aim to explain and remove.
The method itself is simple. Let $\theta_F$ be the fusion parameters, meaning the cross-attention projections and gates. Let $g_t$ and $g_r$ be the task and reconstruction gradients restricted to $\theta_F$. When $g_t \cdot g_r < 0$, we replace the reconstruction gradient on those parameters with
$$\tilde g_r = g_r - \frac{g_r \cdot g_t}{\|g_t\|^2}\, g_t,$$
so that reconstruction can never push the fusion module against the task. Gradients on the encoders and decoder are summed as usual. Projecting in only one direction is deliberate, because reconstruction is the auxiliary objective. We also run a variant that uses the bounded update of Conflict-Averse Gradient Descent for Multi-task learning on $\theta_F$ instead of the projection.
Architectural generalization serves as the evaluation constraint. We train on a two-by-two grid of visual encoder (a ResNet versus a ViT) and audio encoder (a convolutional versus a transformer spectrogram encoder). We add a fifth configuration in which a single shared transformer handles both modalities, to reproduce the setting where gradient harmonization worked. In every configuration we log the gradient angle between the task and reconstruction losses module by module throughout training. The localization hypothesis predicts the same picture in all five configurations: negative angles in the fusion module and benign angles elsewhere.
Baselines
The naive reference point is the same cross-attention model with no decoder, trained on the task loss alone. This is essentially our CMBS reimplementation. For context, CMBS reports 79.7% supervised and 76.0% weakly supervised accuracy on AVE with ResNet-151 features (CMBS). We reproduce it under our encoders rather than relying on those numbers, so that every row shares data, features and schedule.
The strongest existing system on AVE in the file is AVE-CLIP, which fuses contrastively pre-trained AudioCLIP features with a multi-window temporal transformer and reports 83.7% supervised accuracy. AVE-CLIP depends on large-scale contrastive pre-training that is separate from our question. We therefore treat it in two ways. First, as an absolute ceiling we quote. Second, as a feature source: we run our method and the baselines on AudioCLIP features with AVE-CLIP's fusion to check that gains persist on top of the strongest model. Beating AVE-CLIP in absolute accuracy is not the claim. The bar there is a gain over AVE-CLIP's own fusion under identical features and schedule, together with improved robustness under corrupted modalities.
The strongest competitor on the actual question, how to balance task and reconstruction, is reconstruction with a weight tuned by grid search separately for each encoder configuration, with a budget of seven weights times three seeds. Our method gets the same total compute but uses one shared weight. Below that sit three global balancers applied to all shared parameters: GradNorm, PCGrad and CAGrad. For reference on their home benchmark, CAGrad reports a 0.20% average per-task drop on NYUv2 against 3.97% for PCGrad. No reported result was found on Lacuna for any of them on AVE or with a reconstruction auxiliary.
The ablations that isolate the key ingredient are of three kinds. Global versus fusion-only projection tests whether locality matters. Projection on the encoders only, which is the wrong locality, serves as a control. A stop-gradient variant blocks all reconstruction gradient into $\theta_F$.
Why not just…?
Why not just tune the reconstruction weight? This is the most serious threat. On any single encoder pair, a well-tuned weight may match us. Our claim is about portability, so the experiment compares two things: the best single weight shared across all five configurations against our method with that same shared weight, and the per-configuration tuned optimum against us. If the tuned optimum wins on average and its best weight is stable across configurations, the fragility reported for Domain Separation Networks does not appear in this setting and our method adds nothing. We will report that outcome.
Why not just run PCGrad or CAGrad on all shared parameters? The harmonization paper's negative result on modality-specific encoders suggests that global surgery discards useful gradient in parameters that do not really conflict. Our module-wise angle logs test this directly, and the global-versus-local ablation measures what it costs. If global CAGrad equals local projection across the grid, locality is irrelevant and the contribution reduces to "use CAGrad here."
Why not just stop the reconstruction gradient from reaching the attention layer? This is cruder but simpler, and it may work. It throws away the part of the reconstruction signal that agrees with the task. That part is exactly what should push attention to keep reading from a starved modality. If stop-gradient matches projection, we adopt it as the recommended recipe and the paper's contribution becomes the localization finding.
Why not just use late fusion? MultiBench found that simple late fusion often beats cross-modal transformers under noise, for example reducing Vision&Touch error from 0.258 to 0.185. We include late fusion, with and without reconstruction, as a row. If it matches our robustness numbers, reconstruction is compensating for fusion brittleness that need not exist.
Proposed Experiments
The first decisive check is the gradient-angle map. Across the five encoder configurations on AVE, the median task-versus-reconstruction gradient angle should be negative in the fusion module and non-negative in the encoders. If conflict instead appears inside the encoders, the hypothesis is wrong and the method should be dropped.
The second check is weight sensitivity. For each configuration we sweep the reconstruction weight over three orders of magnitude and count the range within one accuracy point of that configuration's best. Our prediction is that fusion-local projection widens this range on every configuration and that a single shared weight comes within one point of each per-configuration optimum.
The third check is robustness, where reconstruction should actually pay off. We test on AVE with audio or visual segments replaced by noise, or zeroed, at test time. We also run weakly supervised AVE, where CMBS reports 76.0%. We expect the gain over the no-decoder model to grow with corruption. For breadth we repeat the sensitivity and robustness protocol on MultiBench's AV-MNIST, CMU-MOSEI and MM-IMDB with its MulT-style cross-attention model; no reported result was found on Lacuna for these datasets in the file.
Finally, the single-shared-backbone configuration tests our explanation of the harmonization paper. Global surgery should work there and fail on separate encoders, while fusion-local projection should work in both. If that asymmetry appears, it accounts for a published negative result. If it does not, the explanation is wrong.
References
- Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization
- Audio-Visual Event Localization in Unconstrained Videos
- Cross-Modal Attention Network for Temporal Inconsistent Audio-Visual Event Localization
- Cross-modal Background Suppression for Audio-Visual Event Localization
- Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognition
- Disentangled Representation Learning for Environment-agnostic Speaker Recognition
- LASPA: Language Agnostic Speaker Disentanglement with Prefix-Tuned Cross-Attention
- Multi-level Disentangling Network for Cross-Subject Emotion Recognition Based on Multimodal Physiological Signals
- Domain Separation Networks
- Conflict-Averse Gradient Descent for Multi-task learning
- AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization
- GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
- Gradient Surgery for Multi-Task Learning
- MultiBench: Multiscale Benchmarks for Multimodal Representation Learning