Where Pseudo-Label Noise Lives: Adapting Only the Channel-Mixing Layers of a Detector

When an object detector trained on clear-weather street scenes is moved to fog, and the original training data cannot be used, the standard fix is self-training. The model labels the new images itself, and a slowly updated copy of the model (the "teacher") supplies those labels to the copy being trained (the "student"). This works until it doesn't. Wrong labels get learned, which makes the next labels more wrong, and accuracy collapses. Practitioners manage the problem with early stopping and careful tuning. We propose a structural answer instead. In modern convolutional backbones such as ConvNeXt, each block splits into two parts: depthwise filters, which look at spatial patterns one channel at a time, and pointwise 1×1 layers, which mix information across channels. Recent evidence says these two parts behave very differently. The spatial filters are generic and interchangeable across datasets. The channel-mixing layers carry nearly all the task-specific, mutually dependent structure that makes networks brittle when pieces are swapped or retrained. Our bet is that pseudo-label noise does its damage by moving generic spatial filters that never needed to move. If so, freezing them during self-training should remove much of the collapse at no cost in peak accuracy.

In two sentences: the parts of a network that are tightly adapted to one another (the pointwise layers, the feature-pyramid connections, the heads) are the only parts a detector needs to adapt to a new domain, and the generic spatial filters should stay fixed. We test whether this restriction makes source-free self-training for detection stable without early stopping. We also test whether the benefit comes from which layers are frozen rather than from simply training fewer parameters.

Core Claim

The claim has two parts, and each can fail on its own. The first is diagnostic. During unrestricted source-free self-training of a ConvNeXt detector, the depthwise filters drift from their source values. That drift is harmful rather than neutral: splicing the source depthwise filters back into a collapsed model should recover accuracy. Splicing source pointwise layers back in should hurt accuracy, because those layers really did need to change. The second part is interventional. Self-training that updates only pointwise layers, normalization parameters, the feature pyramid (the multi-scale neck that merges backbone stages) and the detection heads should reach at least the peak accuracy of unrestricted self-training. It should also stay near that peak for far longer, degrading much less over a long training schedule. It should beat a random frozen subset with the same parameter count, and it should beat the reverse choice of freezing pointwise layers and adapting depthwise ones.

Grounding in Prior Work

Three papers motivate the diagnostic. How transferable are features in deep neural networks? cut AlexNet-style classifiers at a layer and retrained the top. Accuracy dropped even when the frozen bottom came from the same task, which showed that neighbouring layers become fragilely co-adapted: each depends on the specific quirks of the other. We borrow its splicing logic but apply it to a detector during adaptation, where that paper studied a classifier after transfer. The Master Key Filters Hypothesis: Deep Filters Are General showed that ConvNeXt-style depthwise filters can be swapped between models trained on man-made and natural ImageNet subsets with zero accuracy loss. It also located co-adaptation in the pointwise layers. That work stayed within classification and never used the split as an adaptation rule under noisy labels, which is our step. Characterizing Plastic Regions in Neural Networks found contiguous blocks of ResNet layers where adaptation is most useful. Targeted updates there reached a head-relative utility of +0.494, against +0.272 for random blocks, and reduced damage to source knowledge sixfold. Its selection unit is a contiguous depth strip chosen by a Fisher-information score, whereas ours is a within-block operator type chosen by architecture. It also reports that densely connected networks show no plastic region, which bounds our claim to architectures with a clean spatial/channel split.

Two papers supply the adaptation setting and constrain the design. Simplifying Source-Free Domain Adaptation for Object Detection showed three things: re-estimating normalization statistics alone is a strong baseline; its Source-Free Unbiased Teacher reaches 45.0% mAP on Cityscapes→Foggy Cityscapes; and fine-tuning box regression matters. It also showed that pseudo-labels fixed once avoid the collapse loop. We therefore keep the heads trainable and treat fixed pseudo-labels as a serious competitor. Contrastive Mean Teacher for Domain Adaptive Object Detectors (51.9% mAP on Foggy Cityscapes) and Unbiased Mean Teacher for Cross-Domain Object Detection (41.7%) both have access to source data. They are reference points for the gap to non-source-free methods, not direct competitors.

Research Program

We build a Faster R-CNN detector with a feature pyramid on an ImageNet-pretrained ConvNeXt-T backbone and train it on clear Cityscapes. We adapt it source-free to Foggy Cityscapes, to KITTI→Cityscapes (a camera and sensor shift), and to Sim10k→Cityscapes (synthetic to real). We add Pascal VOC→Clipart1k as a style shift, where physical fog simulation cannot help. The adaptation loop follows Source-Free Unbiased Teacher so that the only variable is which parameters receive gradients.

The diagnostic runs first. We run unrestricted self-training well past its usual early-stopping point and save checkpoints along the way. At each checkpoint we splice one parameter group back to its source value and re-measure target mAP. The groups are depthwise filters per stage, pointwise layers per stage, normalization, the feature pyramid and the heads. We also record how far each group has drifted from its source weights and correlate the drift with the pseudo-label error rate on held-out labeled target images.

The intervention follows. We compare adaptation schedules that differ only in which groups are trainable. To check that the effect is about the spatial/channel split, we repeat the study on a ResNet-50 backbone, which has no such split. There the natural analog is the plastic-region selection, so the two selection principles meet on equal ground.

Baselines

The do-nothing reference is the source-trained detector evaluated directly on the target. Just above it is the same detector with normalization statistics re-estimated on target images. ConvNeXt uses layer normalization, which keeps no running statistics, so the ResNet control gets true statistics re-estimation. The ConvNeXt variant instead tunes only the normalization scale and shift parameters on pseudo-labels. Any restricted scheme that cannot clear this bar is pointless.

The strongest existing source-free method is Source-Free Unbiased Teacher from Simplifying Source-Free Domain Adaptation for Object Detection, reported at 45.0% mAP on Cityscapes→Foggy Cityscapes. That number was obtained with the paper's own backbone, so we re-run it on our ConvNeXt-T detector with the same pseudo-label threshold, teacher momentum and augmentation. We give it the same number of hyperparameter trials as our method, including a tuned early-stopping point that uses a small labeled target validation split. The same split is disclosed and used for every method. Its fixed-pseudo-label variant is run under identical conditions. Beating these baselines means two things. Our best-checkpoint mAP must be within noise of, or above, Source-Free Unbiased Teacher's best checkpoint, averaged over three seeds. And our final-checkpoint mAP after a schedule three times the usual length must be clearly higher than its final-checkpoint mAP. The degradation from peak to final is the headline number. The paper does not report KITTI and Sim10k results in a form we can use, so there is no reported result found on Lacuna for source-free Sim10k→Cityscapes or KITTI→Cityscapes with this method; our re-runs supply both.

Full-network adaptation with a starting-point penalty (a weight penalty that pulls parameters back toward their source values) is the classical way to limit drift, and we tune its strength over the same budget. The plastic-region method from Characterizing Plastic Regions in Neural Networks selects a Fisher-chosen contiguous strip of blocks plus heads. It is the strongest existing principle for choosing where to adapt.

Two ablations isolate the key ingredient. The first freezes a random subset of backbone weights matched in count to the depthwise filters. The second is the reverse split: freeze pointwise layers, adapt depthwise ones. The claim holds only if pointwise-only adaptation beats both.

Why not just…?

Why not just adapt the normalization layers? This is the most serious threat. The source-free detection study found normalization re-estimation to be a strong starting point on its own, and normalization-only tuning has very few parameters for noise to corrupt, so it may be just as stable. The difference is expressiveness. Normalization only rescales channels, while pointwise layers can re-mix them, which a style shift like VOC→Clipart plausibly needs. The comparison settles it. If normalization-only tuning matches pointwise adaptation on peak and final mAP across all four shifts, the co-adaptation story adds nothing, and we will report that normalization is sufficient.

Why not just fine-tune everything with a starting-point penalty and stop early? How transferable are features in deep neural networks? showed that full fine-tuning repairs co-adaptation. RIFLE, however, found that such fine-tuning starves early layers of gradient, and it improved on penalized fine-tuning by 0.5–2% in classification. Early stopping is fragile in the source-free setting, because it needs labeled target data that the setting is supposed to lack. We tune this baseline generously. If it matches us, our method's value reduces to not needing a stopping rule, and the degradation metric measures exactly that.

Why not just fix the pseudo-labels once? The fixed-label variant in Simplifying Source-Free Domain Adaptation for Object Detection removes the feedback loop entirely but gives up iterative refinement, and the dynamic teacher reached higher peak accuracy there. Our restriction is compatible with fixed labels. We therefore also run pointwise-only adaptation on fixed labels, to test whether the two sources of stability add up or are redundant.

Why not just simulate fog? Semantic Foggy Scene Understanding with Synthetic Data raised car detection AP in real fog from 30.5% to 35.3% using physics-based fog synthesis. It needs depth maps and a physical model of the shift, and it has no counterpart for clip-art styles or camera changes. It addresses a different lever and can be combined with ours. Including Clipart1k and KITTI keeps our conclusions from depending on fog.

Decisive Checks

The diagnostic must come out first. If, in unrestricted self-training, splicing the source depthwise filters back in neither changes accuracy nor tracks pseudo-label error, then depthwise drift is harmless and freezing it cannot help. In that case we stop and report the negative result, which is itself informative about where noise accumulates in detectors. If the diagnostic holds, the intervention must pass three tests on at least three of the four shifts: pointwise-only adaptation matches Source-Free Unbiased Teacher's peak, it has markedly smaller peak-to-final degradation, and it beats both the random matched-count subset and the reverse split. Finally, the ResNet-50 control should show the plastic-region strip and our operator-type rule converging on similar layers or similar stability. If our rule wins only on ConvNeXt, that confirms the effect is tied to the spatial/channel split rather than being a generic benefit of freezing.