Reliance-Steered Latent Augmentation for Gated Fusion

Many networks combine two or more streams of features: image and text, image and a table of patient attributes, or a local and a global branch inside a vision model. Increasingly they do this with a learned gate, a small network that looks at both streams and outputs soft weights saying how much to trust each one. These gates have a known failure. Early in training one stream usually lowers the loss faster. The gate then saturates toward that stream, the other stream stops receiving gradient, and the model ends up effectively single-stream. It looks fine on clean test data and breaks badly when the favored stream is noisy or missing.

We propose a cheap training-time fix built from latent augmentation, meaning perturbing a network's internal feature vectors rather than its raw inputs. The idea is to add label-preserving noise to each stream's features just before the gate, with the noise scaled by how much the gate currently relies on that stream. The more the model leans on one stream, the harder that stream is shaken along directions of natural within-class variation. The gate is then forced to extract evidence from the neglected stream on those samples. No generator network is involved and no stream is ever deleted outright, so the gate can still settle on an uneven weighting when one stream really is better.

Core Claim

Scaling the perturbation by reliance on each stream, and aiming it along class-covariance directions, prevents gate collapse better than the same perturbation applied uniformly, and better than randomly dropping a stream. It also improves robustness to test-time corruption of the dominant stream, without losing clean accuracy. It should do this at a small fraction of the cost of learned multimodal latent augmentation.

Concretely, let $z_b$ be the feature vector of stream $b$ for a sample of class $c$, and let $\Sigma_{b,c}$ be the class-conditional covariance of stream-$b$ features. This covariance is estimated online, as in Implicit Semantic Data Augmentation for Deep Networks, which showed that such directions encode meaningful within-class change such as pose or background. Let $\bar\alpha_b$ be a running average of the gate's weight on stream $b$ for that sample, with gradients blocked. Let $B$ be the number of streams and $\lambda$ a global strength. We sample

$$\tilde z_b = z_b + \epsilon_b,\qquad \epsilon_b \sim \mathcal{N}\!\left(0,\ \lambda\, \max(0,\ B\bar\alpha_b - 1)\, \Sigma_{b,c}\right).$$

The factor $\max(0, B\bar\alpha_b - 1)$ is zero when the gate splits its weight evenly across streams. It grows only for a stream the gate over-trusts. The perturbed features pass through the gate and the prediction head only, with the usual task loss. The backbones run once, so drawing several samples per example costs little.

Research Program

The work sits between two lines of prior work. Each supplies half the mechanism and leaves the other half open.

Latent augmentation supplies the perturbation. Three prior methods perturb hidden features without regard to which stream or channel the model actually depends on. Manifold Mixup interpolates hidden states, ISDA perturbs features along class covariance, and LEAP transfers angular variance from head classes to tail classes. We keep the covariance direction from ISDA and add a reliance-dependent magnitude.

Local Mixup constrains the design. It shows that perturbations ignoring the geometry of the data create contradictory virtual points, which is why we avoid isotropic noise and cross-class mixing.

The closest joint work is Learning Multimodal Data Augmentation in Feature Space (LeMDA). It trains an adversarial variational autoencoder to perturb joint multimodal features, then merges them with static late fusion. It reports reduced training throughput and difficulty scaling to earlier layers. Our method differs on three points: it is generator-free, it targets a gated fusion module, and its perturbation is driven by the gate's own weights rather than by an adversary.

Adaptive fusion supplies the gate and the failure we target. Attentional Feature Fusion replaces addition with multi-scale channel attention gates (AFF, and its two-stage iterative version iAFF). It never studies gate behavior under perturbation.

No Data? No Problem (RoVTL) is the main motivating evidence that gates need explicit protection against a dominant stream. It uses gated cross-attention, gradient decoupling and a monotonicity loss. Its only augmentation is random masking of tabular attributes, and its gate acts in one direction only. We ask whether a single continuous, reliance-aware perturbation can do the work those extra losses do.

MaskFusion calls its instance-adaptive feature injection "augmentation", but it does not expand the training distribution. It is therefore an architectural relative, not a competitor.

We will build the method in two settings. The first is two-stream late fusion with a gate on the LeMDA benchmarks, Hateful Memes (image and text) and Petfinder (image, text and tabular). The second is within-network fusion, using iAFF-ResNet on CIFAR-100, where the gate arbitrates between branches of a single network.

For each run we log three things over training. The first is gate saturation: the fraction of samples whose largest stream weight exceeds 0.95. The second is the accuracy of a linear probe trained on each stream's features alone, which shows whether the neglected stream is still learning. The third is the variance of the gate weights across samples. A known risk is feedback: perturbing a stream lowers its gate weight, which in turn reduces the perturbation. Blocking gradients through $\bar\alpha_b$ and using a slow running average should damp this, and we will report whether the weights oscillate.

Baselines

The naive reference point is the same gated architecture trained with its standard input augmentation and no latent perturbation. This shows how much collapse occurs by default.

A second, static-fusion reference is concatenation with Manifold Mixup applied to the fused vector. This asks whether a gate is needed at all. The file reports Manifold Mixup at 18.35% CIFAR-100 error with PreActResNet-34, against 23.55% for vanilla training (Verma et al.). We will rerun it on our backbone rather than compare against that number directly.

The strongest existing method on the multimodal benchmarks is LeMDA. The file reports only a "~6% accuracy increase" over standard multimodal baselines on Hateful Memes and Petfinder, with no absolute value. We will therefore rerun it from the authors' description on the same backbones, data splits and epoch budget. We will give it the same number of hyperparameter trials as our method and report wall-clock time alongside accuracy. Beating LeMDA means matching or exceeding its clean accuracy while clearly exceeding it under test-time corruption of the dominant stream, at lower training cost. Matching its accuracy at a fraction of the cost would already be a useful result.

On CIFAR-100 the strongest gated reference is iAFF itself. The file gives only an ImageNet figure for it: iAFF-ResNet-50 at 20.2% top-1 error (Dai et al.). For iAFF on CIFAR-100 there is no reported result found on Lacuna, so we will train it ourselves. We will also add ISDA to iAFF as a strong latent-augmentation combination. ISDA alone reports 21.9% ImageNet error with ResNet-50 (Wang et al.).

Four baselines isolate the key ingredient. Uniform ISDA-style perturbation of every stream before the gate tests whether steering by reliance matters. Its total perturbation energy is matched to ours on average. Modality dropout at a matched rate tests continuous perturbation against hard removal. For this there is no reported result found on Lacuna on these benchmarks; RoVTL's tabular masking is the nearest relative. An entropy penalty on the gate weights tests whether any pressure against saturation suffices. Inverse steering, which perturbs the less-trusted stream more, is a sanity check that should fail to prevent collapse.

Why not just…?

Why not just drop the dominant stream at random during training? This is the most serious threat. RoVTL's success with attribute masking suggests removal-style augmentation can work. Dropout, however, gives the gate an all-or-nothing signal. It teaches the model to survive a missing stream, not a degraded one. It also presents the stream in a state never seen at test time unless data is actually missing. If matched-rate modality dropout equals our method on both corruption robustness and clean accuracy, the steering adds nothing and we will say so. The comparison is run on both kinds of corruption: noisy streams and missing streams.

Why not just put an entropy or load-balancing penalty on the gate? Such a penalty pushes weights toward even splits whether or not that is correct. Attentional Feature Fusion and RoVTL both show that uneven, content-dependent weighting is exactly what makes gating useful. Our perturbation leaves a stream's weight high if that stream stays informative under label-preserving variation. The penalty baseline will show whether this distinction matters, as measured by per-sample gate variance and clean accuracy.

Why not just apply ISDA or Manifold Mixup uniformly and keep the gate? This is the direct ablation. If uniform perturbation matches reliance-steered perturbation at equal energy, the contribution reduces to "latent augmentation also helps gated models". We would report that as a smaller finding rather than claim the steering mechanism.

Why not just use LeMDA with a gated fusion head? It is a plausible combination. But its adversarial generator is the cost bottleneck its authors identify, and the adversary has no notion of which stream the gate over-trusts. We include LeMDA plus a gate as an extra comparison so that the cost-versus-robustness trade-off is measured directly.

Proposed Experiments

On Hateful Memes and Petfinder we measure three outcomes: clean accuracy; accuracy when the dominant stream is corrupted at test time with feature noise, input corruption or removal; and training throughput. On CIFAR-100 with iAFF-ResNet we measure error, calibration error and accuracy when one branch is corrupted. All methods share backbones, splits, epochs and tuning budgets, and results are averaged over at least three seeds.

Three checks decide the claim. First, reliance-steered perturbation must reduce gate saturation and raise the neglected stream's probe accuracy relative to uniform perturbation at equal energy, and inverse steering must not do so. Second, it must beat matched modality dropout on robustness to corrupted but present streams, without losing on missing streams. Third, it must retain LeMDA-level clean accuracy at substantially lower training time. If the first check fails the mechanism is wrong. If only the second fails, the practical answer is modality dropout, and the paper becomes a careful negative result about steering.