When a model learns from two input streams at once, such as video and audio, it usually leans on whichever stream is easier and starves the other. That other stream's encoder ends up barely trained. Separately, training large encoders end to end runs out of GPU memory, because ordinary backpropagation keeps every layer's intermediate outputs (its "activations") until the backward pass. Tricks that save this memory either rebuild those activations exactly from the layer outputs or store a cheap approximation of them, and the approximations make gradients less accurate. This proposal argues that the two problems should be handled together. The dominant modality is the one whose gradients balancing methods already shrink and deliberately add noise to. Storing its activations lossily therefore costs almost nothing. The saved memory should go to computing exact gradients for the weak modality, which is the one whose small and fragile gradient signal actually decides whether balancing works.
The core claim has two parts. The first is a hypothesis we will test: approximate memory-saving schemes, applied uniformly, damage the suppressed modality's gradient far more than the dominant one's, and so quietly make modality dominance worse. The second is a method: an allocation rule, steered online by the same per-batch contribution signal that balancing methods already compute. It stores 1-bit approximations for whichever modality currently dominates, and exact or exactly reconstructible activations for the other. If both parts hold, the result is end-to-end joint training of both encoders at roughly the memory of compressed training, with the balance of exact training.
Where this sits in prior work
Three papers motivate the problem. Balanced Multimodal Learning via On-the-fly Gradient Modulation (OGM-GE) measures, per batch, how much each modality contributes to the prediction. It shrinks the dominant modality's gradient by $1-\tanh(\alpha\rho)$, where $\rho$ is the contribution ratio and $\alpha$ a strength hyperparameter, and then injects Gaussian noise to restore stochasticity. What Makes Training Multi-Modal Classification Networks Hard? (G-Blend) shows that modalities overfit at different rates, and that a naive joint model can lose to the best single modality. See-Saw Modality Balance (BALGRAD) reweights and projects gradients in vision-language models, but freezes the ViT and BERT encoders and trains only the fusion and classifier layers. That choice leaves untouched the under-trained encoders OGM-GE identifies as the root cause. Our proposal does not invent a new balancing signal; it reuses OGM-GE's contribution ratio.
Three papers supply the memory-saving machinery, and none of them was built for multimodal inputs. The Reversible Residual Network splits each layer's input into two halves joined additively, so inputs can be recovered from outputs by subtraction and nothing needs storing. Make Pre-trained Model Reversible (MEFT) wraps pretrained blocks in such reversible containers. It notes that 16-bit reconstruction drifts, which pushes it to accumulate in 32-bit. Learning with Auxiliary Activation for Memory-Efficient Training (AAL) stores a proxy instead of the true activation, either a 1-bit sign mask or the reused residual input. It proves convergence as long as the proxy gradient keeps a positive "learning indicator", meaning a positive alignment with the true gradient. All three were evaluated only on single-modality vision or language models.
The invertibility literature constrains which components are exact for free. Exact reconstruction needs one-to-one operations. In-Place Activated BatchNorm must swap ReLU for LeakyReLU for this reason, and ReTAL had to move activations inside residual sub-blocks to make pretrained video backbones reversible. ReLU-style layers can be recovered exactly only with an extra 1-bit mask of which units were active. We use these facts to define the "exact" tier for each encoder type.
We also correct one framing that the intersection invites. One might think gradient modulation is incompatible with reversible backpropagation, since reversible networks rebuild activations during the backward pass. It is not incompatible. Reconstruction uses only forward states and weights, so scaling or adding noise to the gradient flowing into a stream leaves it untouched. The real interaction is the other direction: reconstruction and compression errors corrupt gradients, and the open question is whose gradients.
Hamiltonian Asymmetric Fusion is the nearest work that combines efficiency with imbalance. It fixes a primary modality by hand and refines only the auxiliary one, and it neither reduces training activation memory nor lets the roles change. We make the roles data-driven, allow them to switch per step, and apply the asymmetry to how exactly gradients are computed rather than to the fusion architecture.
Core Claim
For modality $m$ at training step $t$, let $g_m$ be the exact gradient with respect to that modality's encoder parameters and $\tilde g_m$ the gradient obtained under a given memory scheme. We track a per-modality version of AAL's learning indicator, the cosine alignment between the two:
$$c_m(t) = \frac{\langle \tilde g_m, g_m \rangle}{\lVert \tilde g_m \rVert\,\lVert g_m \rVert}.$$
Our hypothesis is that under uniform compression, $c_m$ is markedly lower for the suppressed modality. The reasoning is that compression error scales roughly with activation statistics, while the weak modality's true gradient is small, so the error-to-signal ratio is worst there. If so, uniform memory saving lowers the weak modality's effective learning rate in exactly the place balancing methods try to raise it.
The allocation rule uses OGM-GE's contribution ratio $\rho$, smoothed over a short window. Whichever modality has $\rho>1$ gets 1-bit sign-mask storage. The other gets exact treatment: stored activations, or reversible coupling with 32-bit reconstruction where an encoder can be wrapped as in MEFT. Activations live for one step only, so switching allocation when dominance flips costs nothing.
A second, more speculative claim concerns noise. OGM-GE injects noise into the dominant modality on purpose. If we measure the variance of the dominant modality's compression-induced gradient error, we can subtract it from the injected noise, so that compression partly substitutes for that noise rather than adding to it.
Research Program
The first stage is diagnostic and must come first, because the method only makes sense if the asymmetry exists. We will train audio-visual models with exact backpropagation as the reference. At regular intervals, we will compute $\tilde g_m$ under each memory scheme on the same batch and log $c_m$ for each modality: AAL sign proxies, AAL residual proxies, MEFT-style reversible wrapping in 16-bit and 32-bit reconstruction, and uniform activation quantization. Over training, we will relate $c_m$ to $\rho$.
The second stage tests whether this gradient corruption turns into representation damage. We will measure linear-probe accuracy of each encoder alone after joint training under each scheme.
The third stage implements the allocation rule on top of OGM-GE and on top of G-Blend. It then compares the result, at matched peak activation memory, against uniform schemes and exact training.
The fourth stage tests noise accounting as an optional refinement. If it does not help, we will report it as a negative result and drop it.
The work uses transformer encoders for both streams, for example ViT-style video and audio-spectrogram encoders over multi-second clips. This is the regime where activation memory actually binds. The small ResNet-18 setups in the OGM-GE line of work fit in memory without any of this machinery.
Baselines
The naive reference is plain concatenation fusion with exact backpropagation and no balancing, at whatever batch size fits. On CREMA-D, OGM-GE reports that concatenation goes from 51.7% to 61.9% accuracy once modulation is added. On Kinetics, G-Blend reports a naive joint model at 71.7%, below RGB-only at 72.6%. These figures show how much room balancing has.
The strongest existing method for the balancing question is OGM-GE, or G-Blend where we use Kinetics-style data, run with exact backpropagation and gradient checkpointing so that it fits the same memory budget. This is the strongest competitor because it is exact: it gives the best balance any memory scheme could reach, and it pays only in wall-clock time. On VGGSound, OGM-GE reports accuracy rising from 49.1% to 50.6%. On Kinetics, G-Blend reports 75.8%. We will rerun both in our encoder setting rather than borrow these numbers, because our backbones differ.
The strongest existing memory method is uniform AAL with sign proxies combined with OGM-GE, which is the natural thing a practitioner would try. For the end-to-end vision-language comparison on UPMC Food-101 and MM-IMDb, the reference is BALGRAD with frozen encoders. BALGRAD reports a 12.5 percentage-point gain for the weak text modality over its baseline on Food-101. No reported raw accuracy was found on Lacuna for MM-IMDb.
The ablation that isolates the key ingredient is reversed allocation: exact storage for the dominant modality and 1-bit storage for the weak one, at identical memory. If our hypothesis is right, reversed allocation should be clearly worse than our allocation. If the two are equal, the asymmetry is not the mechanism.
All methods share the same data, encoders, number of steps, and a fixed hyperparameter search budget for $\alpha$ and the learning rate. Beating the baselines means two things. First, matching exact checkpointed OGM-GE accuracy within seed noise, and matching its weak-modality probe accuracy, at the memory of uniform compression with lower wall-clock time than checkpointing. Second, significantly exceeding uniform AAL plus OGM-GE on weak-modality probe accuracy.
Why not just…?
Why not just use gradient checkpointing with OGM-GE? This is the real threat. It is exact, works with any architecture, and leaves balancing untouched. AAL reports that checkpointing requires an extra forward pass and so slows training, whereas its proxies give large compression with little slowdown. Even so, if checkpointing's overhead in our setting turns out to be modest, our method's advantage reduces to speed. We will report accuracy, peak memory, and time per step for both at matched memory. If checkpointing wins on accuracy at acceptable cost, the surviving contribution is the diagnostic finding about which memory schemes harm the weak modality.
Why not just compress everything uniformly and let OGM-GE compensate? OGM-GE amplifies the weak modality relative to the dominant one, but amplifying a gradient whose direction is wrong does not fix it, because a low $c_m$ is a direction error, not a scale error. The comparison between uniform compression and our allocation, together with the reversed-allocation ablation, settles this directly.
Why not just freeze the encoders, or use low-rank adapters, as BALGRAD does? MEFT shows that adapter methods reduce trainable parameters but still cache activations through every frozen layer, so they do not remove the memory problem. Freezing also leaves the lazy encoders that OGM-GE blames for dominance. Our end-to-end comparison against frozen BALGRAD on Food-101 and MM-IMDb tests whether unfreezing at low memory actually helps the weak modality.
Why not just use modality dropout? Wolf et al. drop whole modalities to force use of the others. G-Blend argues that generic regularizers do not fix differing overfitting rates. Dropout also saves no memory. We include it as an extra balancing baseline under exact checkpointed training.
Proposed Experiments
The first decisive check is the diagnostic itself, run on CREMA-D and Kinetics-Sounds. Does $c_m$ under uniform sign proxies or 16-bit reversible reconstruction fall significantly further for the low-$\rho$ modality than for the high-$\rho$ one, consistently across seeds? If not, we stop and report that memory-efficient backpropagation is modality-neutral. That outcome would itself be useful to anyone combining these methods.
If the asymmetry appears, the second check runs on CREMA-D, Kinetics-Sounds, and VGGSound. It compares our allocation, reversed allocation, uniform compression, and exact checkpointed training, all with OGM-GE at matched peak activation memory. We report overall accuracy, per-encoder probe accuracy, time per step, and $c_m$ over training.
The third check extends to unfrozen ViT and BERT training on UPMC Food-101 and MM-IMDb, where text and image swap dominance across datasets. This tests whether allocation tracks the correct modality without manual role assignment, which is the limitation HAF leaves open.
A final stress test lowers reconstruction precision in the exact tier to 16-bit. This tells us whether the weak modality is uniquely sensitive to reconstruction drift, the failure MEFT reports in single-modality models.
References
- Balanced Multimodal Learning via On-the-fly Gradient Modulation
- What Makes Training Multi-Modal Classification Networks Hard?
- See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias
- The Reversible Residual Network: Backpropagation Without Storing Activations
- Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning
- Learning with Auxiliary Activation for Memory-Efficient Training
- In-Place Activated BatchNorm for Memory-Optimized Training of DNNs
- ReTAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization
- Hamiltonian Asymmetric Fusion: One-Way Safe Directed Refinement under Modality Imbalance
- ModDrop: adaptive multi-modal gesture recognition