Shielded Gradient Reversal: Adversarial Domain Adaptation That Spares Skill Neurons

When a vision model trained on one kind of image, such as product photos, is deployed on another, such as clip art, its accuracy drops. The standard remedy, adversarial domain adaptation, trains a second network to guess which domain each feature came from. It then pushes the main network to make that guess impossible, so its features stop revealing the domain. This push is applied uniformly to every unit in the representation. Separately, interpretability work shows that trained networks are not uniform. A small fraction of individual units carries most of the task-relevant signal, and silencing those few units damages the task far more than silencing random ones. Our proposal joins these observations. We identify which units encode the domain and which encode the task. The domain-confusion gradient is then applied only to the former, and the task-critical units are shielded from it.

We also make two further claims. First, a known failure of adversarial adaptation is that features collapse onto one dominant direction and lose fine class distinctions. We conjecture that this collapse is, at least in part, the erasure of task-critical units, and we test this directly rather than assume it. Second, the method is defined only in terms of per-unit activation statistics, so the same code should run on a convolutional network and on a Vision Transformer. Existing adaptation modules that work well on transformers are tied to the attention mechanism and cannot do this.

Core Claim

Adversarial alignment hurts class discrimination because its gradient degrades a small set of task-predictive units, and the representation needs those units to stay intact. Two things follow if this is right. Restricting the gradient-reversal signal to units that are domain-informative but weakly task-predictive should keep target accuracy gains while avoiding the collapse. And because "unit" can mean a pooled convolutional channel or a hidden neuron in a transformer's feed-forward layer, one selection rule should transfer across both architectures without redesign.

Relation to Prior Work

Three lines of work motivate the claim. Finding Skill Neurons in Pre-trained Transformer-based Language Models scores each feed-forward neuron by how well its activation alone predicts task labels. It finds that perturbing the top-scoring few percent breaks the task while perturbing random units does not. Combination Generalization of Capability-Specific Neurons in LLMs reports that disabling 10% of capability neurons costs 21.7% versus 0.1% for random units. It also finds that fine-tuning only those neurons, about 0.05% of parameters, beats tuning everything else. Both papers study language models with no domain shift. We borrow their scoring idea and apply it to vision backbones under adversarial training.

On the adaptation side, Transferability vs. Discriminability: Batch Spectral Penalization for Adversarial Domain Adaptation shows that adversarial training inflates the top singular value of the feature matrix and suppresses the secondary directions that separate classes. It repairs this with a penalty on the largest singular values. That paper supplies our main diagnostic and our strongest same-backbone competitor. It treats the symptom at the level of the whole feature matrix and does not ask which units are being damaged.

The closest existing method is IDANI: Inference-time Domain Adaptation via Neuron-level Interventions. At test time it ranks neurons by the gap between source and target mean activations, then shifts the most divergent ones toward source statistics. That simple mean-gap ranking beat a learned linear probe in 47 of 52 domain pairs. We reuse its ranking, but we do so inside training, within an adversarial objective, and with an explicit task-predictivity filter. IDANI has none of these, and it was only evaluated on text encoders.

Time Series Domain Adaptation via Channel-Selective Representation Alignment is the strongest evidence that selective alignment matters. Aligning all input channels uniformly collapsed to 0.46 macro F1 under sensor saturation, while channel-selective alignment kept 0.922. However, it selects raw input sensors, uses optimal transport rather than an adversarial game, and never looks inside the network. Selection at the gradient level, in the style of Improving Adversarial Transferability via Neuron Attribution-based Attacks and DANAA, shows that neuron attribution can find units that behave consistently across architectures. Both papers use this to attack models, not to adapt them.

On architecture, TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation re-weights self-attention by patch-level domain scores. It ignores feed-forward units and cannot run on a CNN. Conversely, Cross-Domain Adaptive Teacher for Object Detection and COT are CNN methods whose authors flag porting to ViTs as an open problem. Function-aware methods such as NeuronMerge and FANG constrain our design. They show that splitting units into general and specific groups prevents interference in merging and pruning, which suggests that a hard partition, rather than a uniform reweighting, is the right primitive.

Research Program

The method has three parts. The first is a backbone-agnostic definition of a unit. In a ResNet, a unit is one output channel of a stage, averaged over spatial positions. In a ViT, a unit is one hidden neuron of a feed-forward block, averaged over tokens. We tap one layer per stage in the CNN and every third block in the ViT.

The second part is a pair of per-unit scores, recomputed every few epochs. The domain score $d_j = |\mu_{S,j} - \mu_{T,j}| / \bar\sigma_j$ follows IDANI's mean-gap ranking, normalized by the pooled standard deviation. Here $\mu_{S,j}$ and $\mu_{T,j}$ are unit $j$'s mean activation on source and target batches, and $\bar\sigma_j$ is its pooled standard deviation. The task score $t_j$ is the skill-neuron predictivity from Wang et al., meaning the best one-versus-rest accuracy of thresholding unit $j$ against source labels. Because labels are only available on the source domain, $t_j$ is computed there.

The third part is shielded reversal. A single domain discriminator reads the concatenated pooled activations from all tapped layers. In the backward pass, the gradient-reversal step at each tapped unit is multiplied by a mask:

$$\frac{\partial \mathcal{L}}{\partial h_j} \leftarrow \frac{\partial \mathcal{L}_{task}}{\partial h_j} - \lambda\, m_j \frac{\partial \mathcal{L}_{dom}}{\partial h_j}, \qquad m_j = \mathbb{1}[d_j > \tau_d]\,\mathbb{1}[t_j < \tau_t].$$

Here $h_j$ is unit $j$'s activation, $\mathcal{L}_{task}$ is the source classification loss, $\mathcal{L}_{dom}$ is the discriminator loss, and $\lambda$ is the usual reversal weight. The thresholds $\tau_d$ and $\tau_t$ are set by quantiles: the top 20% of units by domain score and excluding the top 5% by task score, with both quantiles swept.

Shielding is not absolute. Upstream weights changed through masked units still affect the shielded ones. For that reason we measure shielding directly instead of assuming it. The task loss uses class-conditional discriminator inputs as in Conditional Adversarial Domain Adaptation, so the method is a drop-in change to CDAN.

Alongside the method we run an analysis study. For plain CDAN on each backbone, we track three quantities over training: the ratio of the top singular value to the sum of the top five, the source-label predictivity of the units that were top-ranked at initialization, and target accuracy. Under the erasure hypothesis, the loss in skill-unit predictivity should precede or co-occur with the singular-value spike and should predict the final target accuracy deficit across domain pairs.

Baselines

The naive reference point is source-only training on each backbone, ResNet-50 and ViT-B/16, with no adaptation. It tells us how much any method gains.

The first adversarial baseline is CDAN with entropy conditioning, run with our own code on both backbones. Conditional Adversarial Domain Adaptation reports 65.8% average accuracy on Office-Home and 87.7% on Office-31 for this method with ResNet-50. Shielded reversal is a one-line modification of CDAN, so this comparison isolates the mask.

The strongest same-backbone competitor on the precise question is CDAN plus Batch Spectral Penalization. It addresses the same failure we target, but with a matrix-level penalty instead of unit selection. Transferability vs. Discriminability reports 66.3% on Office-Home, 88.5% on Office-31, and 75.9% on VisDA-2017 with ResNet-50.

On ViTs the strongest reported method is TVT. TVT reports 83.6% on Office-Home and 86.7% on VisDA-2017. On VisDA, COT reports 89.7% with a ResNet backbone in COT. These are not on matched backbones or recipes, so we rerun TVT from its description on our ViT-B/16 and rerun COT on our ResNet-50.

All methods share the same pretrained checkpoints, augmentation, number of training steps, and a fixed tuning budget of 20 configurations per method. Model selection follows a protocol that does not peek at target labels: either importance-weighted source validation, or reporting the last checkpoint. We use the same protocol for everyone.

Beating the baselines means three things. On ResNet-50 we must exceed our rerun of CDAN+BSP on the Office-Home and VisDA averages by more than the seed standard deviation over three seeds. On ViT-B/16 we must match or exceed our rerun of TVT without using any attention-specific module. And one hyperparameter setting, transferred unchanged, must win on both backbones.

The key ablations hold the number of masked units fixed. A random mask of equal size tests whether selection matters at all. A domain-score-only mask without the task filter tests whether shielding skill units is the active ingredient. An inverted mask, which aligns the high task-score units, should hurt if the hypothesis is right. A shield-only variant applies the task filter but aligns every other unit, which tests whether the domain filter matters. Per-unit moment matching on the same selected units, with no discriminator, tests whether the adversarial game adds anything.

Why not just…?

Why not just add Batch Spectral Penalization to CDAN? This is a real competitor, because it already restores the secondary directions that adversarial training suppresses. It reported gains of roughly half a point to two points over CDAN on the ResNet-50 benchmarks in the file. Our bet is different: the damage is concentrated in specific units, so protecting them should beat redistributing variance across the whole matrix. A spectral penalty can also restore variance in directions that carry domain information rather than class information. The analysis study settles this. If BSP recovers skill-unit predictivity as fully as shielding does and matches it in accuracy, then the unit-level story adds explanation but no method, and we will report that. We also test whether the two combine additively, which would indicate they fix different things.

Why not just re-standardize each unit's target statistics, as IDANI does at test time? If domain shift lives mainly in per-unit means and variances, a moment-matching step is cheaper than any adversarial game and could match it. IDANI's large text gains suggest this happens. This is the most serious threat, and the moment-matching ablation on the same selected units addresses it directly. Its weakness is that it treats units as independent dials, while NeuronMerge documents strong non-linear interactions in feed-forward layers that additive shifts ignore. The adversarial discriminator can also detect shift in joint and higher-order structure. If moment matching ties shielded reversal on these benchmarks, the honest conclusion is that the shift is first-order and the selection rule, not the adversary, is the contribution.

Why not just use a ViT with TVT and forget CNNs? TVT's numbers are strong, but its mechanism requires attention maps, and it says nothing about feed-forward units, where the skill-neuron literature locates task signal. Our question is whether one unit-level rule works across architectures, which TVT cannot answer by construction. Practically, if our ViT result falls well short of TVT, the cross-architecture claim is too weak to matter. In that case we would also test shielding as an add-on to TVT, to see whether it helps there.

Why not just lower the reversal weight? Weaker uniform alignment also reduces collapse. Our tuning grid includes λ for CDAN, so the CDAN baseline is already its best-λ version. The random-mask ablation, which also weakens total alignment by the same amount, separates "less alignment" from "better-targeted alignment."

Proposed Experiments

The main benchmarks are Office-31, Office-Home, and VisDA-2017, using all transfer pairs on the first two. We run ResNet-50 and ViT-B/16 with ImageNet pretraining, plus ConvNeXt-T as a third, held-out architecture. On ConvNeXt we apply the quantile settings chosen on the other two without retuning. The primary metric is average target accuracy over three seeds.

The secondary metrics carry the scientific claim. They are the retention of initial skill-unit predictivity after adaptation, the top-singular-value ratio, and the fraction of the discriminator's final accuracy that the shielded units alone can still achieve. The last metric tells us how much domain information we deliberately left in place.

Three results would be decisive. First, across domain pairs under plain CDAN, the drop in skill-unit predictivity should correlate with the target accuracy deficit relative to the best method. Second, shielded reversal should reduce the singular-value spike without any spectral term. Third, the domain-score-only and random masks should fall back to CDAN-level accuracy, while the full mask does not. If shielded units retain predictivity but accuracy does not improve, the erasure hypothesis is wrong and the collapse has another cause. If the inverted mask performs as well as the full mask, unit selection is not doing what we claim.

A smaller follow-up applies the same masked reversal to the Adaptive Teacher detector on Foggy Cityscapes, where Cross-Domain Adaptive Teacher reports 50.9% mAP on a CNN. The test is whether the method ports to a detector with a ViT backbone, which that paper left open. This is exploratory, and we do not count it as evidence for the main claim.