Abstract
Deep learning models often struggle with real-world distribution shifts such as noise, blur, and weather conditions. This paper introduces Gain From Neighbors (GFN), a novel training methodology designed to enhance model robustness by focusing on "near-miss" classification errors. Unlike standard adversarial training that pushes samples away from the ground truth in arbitrary directions, GFN specifically perturbs inputs toward semantically similar "neighboring" classes. Combined with an inter-class distance-weighted loss, GFN encourages models to learn more distinct decision boundaries between similar categories. Experimental results across ImageNet variants and object detection benchmarks demonstrate that GFN significantly improves robustness and clean accuracy without adding architectural complexity.
The Fragility of Deep Learning
While modern neural networks achieve impressive performance on clean datasets, they remain notoriously fragile when deployed in the wild. Common environmental factors—motion blur, fog, or low-light conditions—can cause catastrophic drops in accuracy. Traditional defenses like data augmentation or standard adversarial training often fail to account for the structured nature of model failures.
The authors observe that when models fail under distribution shift, they typically misclassify an image into a semantically similar category rather than a random one. For instance, a "beacon" might be mistaken for a "drilling platform." This suggests that while the model captures the general essence of the image, it lacks the fine-grained discriminative power to separate closely related classes under noise.
The GFN Methodology
To address this, the researchers propose Gain From Neighbors (GFN), which consists of two core components:
1. Neighboring Gradient Perturbation
Standard adversarial training (AT) generates perturbations to maximize loss, pushing the sample away from the ground truth label in any direction. In contrast, GFN utilizes gradient priors to steer perturbations specifically toward the most similar neighboring classes. This approach simulates the most probable real-world errors, forcing the model to confront and resolve ambiguities between similar concepts.
2. Inter-Class Distance Weighted Loss
To further refine the feature space, GFN introduces a novel loss function that incorporates inter-class similarity. The loss assigns higher penalties to misclassifications involving classes that are semantically close. By penalizing "near-misses" more heavily, the model is incentivized to create wider margins and more dispersed feature representations in the embedding space.
Key Insights: Why Neighbors Matter
Statistical analysis of model errors reveals that most mistakes under distribution shift fall within the top 100 most similar classes. This confirms that robustness is fundamentally linked to the model's ability to maintain inter-class distance. By comparing standard Vision Transformers (ViTs) to robust models like DINOv2, the authors found that robust models naturally exhibit larger distances between class embeddings. GFN effectively "teaches" this property to standard models during fine-tuning.
Furthermore, GFN is computationally efficient. By constraining the search space for perturbations to neighboring regions, the optimization landscape becomes less complex, allowing the model to converge faster than with traditional adversarial training.
Experimental Results
The GFN framework was evaluated across a diverse set of benchmarks for both classification and object detection.
Classification Performance
On ImageNet-C (corruptions), GFN achieved a corruption robustness (mCE) of 41.4% using a ViT-B backbone, a 2.3% improvement over the previous state-of-the-art method, TAPADL. Notably, GFN avoids the typical "robustness-accuracy trade-off"; it improved clean top-1 accuracy on ImageNet from 76.3% to 79.9%.
Robustness in the Wild
In object detection tasks, GFN showed significant gains on real-world challenging datasets:
- ExDark (Low-light): Improved mean Average Precision (mAP) to 54.3.
- RTTS (Foggy): Demonstrated superior performance compared to specialized robust architectures like FAN.
Conclusion
GFN provides a simple yet powerful strategy for boosting model robustness by leveraging the semantic structure of the label space. By focusing on the most likely failure modes—confusion between similar classes—GFN builds models that are not only more stable under noise but also more accurate on clean data.
Summary updated