Abstract
Building object detectors that are robust to domain shifts is critical for real-world applications. Prior approaches fine-tune a pre-trained backbone and risk overfitting it to in-distribution (ID) data and distorting features useful for out-of-distribution (OOD) generalization. We propose to use Relative Gradient Norm (RGN) as a way to measure the vulnerability of a backbone to feature distortion, and show that high RGN is indeed correlated with lower OOD performance. Our analysis of RGN yields interesting findings: some backbones lose OOD robustness during fine-tuning, but others gain robustness because their architecture prevents the parameters from changing too much from the initial model. Given these findings, we present recipes to boost OOD robustness for both types of backbones. Specifically, we investigate regularization and architectural choices for minimizing gradient updates so as to prevent the tuned backbone from losing generalizable features. Our proposed techniques complement each other and show substantial improvements over baselines on diverse architectures and datasets.
Introduction: The Challenge of Backbone Distortion
In modern computer vision, the standard practice involves taking a massive model pre-trained on ImageNet (the backbone) and fine-tuning it for downstream tasks like object detection. While this approach yields high performance on data similar to the training set (In-Distribution or ID), it often fails when the environment changes—such as moving from clear weather to foggy conditions or artistic renditions (Out-of-Distribution or OOD).
The authors argue that "naive" fine-tuning is often too aggressive, forcing the backbone to overwrite its general-purpose, robust features with task-specific ones that overfit the training data. This phenomenon, termed backbone distortion, is the primary barrier to OOD robustness.
Quantifying Vulnerability with Relative Gradient Norm (RGN)
To address this, the researchers introduce Relative Gradient Norm (RGN), defined as the ratio of the gradient norm to the parameter norm. RGN serves as a "vulnerability meter": a high RGN indicates that a layer is likely to change drastically during training, leading to feature distortion.
As illustrated in Figure 2, architectures exhibit vastly different distortion profiles. ResNet50 shows high RGN across all layers, making it highly vulnerable. In contrast, newer architectures like ConvNeXt and EfficientNet have significantly lower RGNs, suggesting they are "stiffer" and naturally preserve their pre-trained features better.
A Recipe for Robustness
The paper proposes a multi-pronged strategy to minimize distortion:
- Decoder-Probing (DP-FT): Instead of tuning the entire network immediately, the backbone is initially frozen while only the "decoder" (the detection-specific layers) is trained. This allows the decoder to stabilize before full fine-tuning begins.
- Weight Regularization (WR): To prevent weights from drifting too far from their robust starting point, a penalty is added to the loss function. Scaling this penalty by the RGN proves highly effective across architectures.
- Squeeze-and-Excitation (SE) Blocks: The authors discovered that SE-blocks act as "gradient masks." By producing sparse activation patterns, they suppress aggressive gradients flowing back to the backbone early in training. Manually inserting SE-blocks into vulnerable backbones like ResNet50 significantly reduces distortion.
Experimental Results and Robustness Gains
The proposed methods were evaluated across diverse benchmarks, including Pascal VOC, Cityscapes, and COCO, along with OOD versions like Foggy Cityscapes and artistic datasets (Watercolor, Comic, Clipart).
As shown in Figure 1, the "Our Tuning" recipe consistently outperforms "Plain Tuning" in OOD scenarios. For instance, on the Pascal-to-Comic task with a ResNet50 backbone, performance jumped from 7.5 mAP to 16.8 mAP. Qualitative results in foggy conditions further demonstrate the model's improved ability to detect objects that standard fine-tuning misses.
Key Insights
- Architecture Matters: Not all backbones distort equally; newer designs are often inherently more robust to fine-tuning.
- The SE-Block Effect: These blocks effectively shield the backbone from being "over-written" by early training errors.
- Longer Warm-up: Increasing the "Decoder-Probing" phase reduces the eventual RGN of the backbone, preventing it from being pulled in the wrong direction.
Conclusion
By "minding the backbone" and using RGN to guide regularization and architectural choices, it is possible to build object detectors that maintain high accuracy while significantly improving robustness to real-world domain shifts.
Code is available at https://github.com/VisionLearningGroup/mind_back.
Summary updated