On the Use of Anchoring for Training Vision Models

Vivek Narayanaswamy, Kowshik Thopalli, Rushil Anirudh, Yamen Mubarka, Wesam A. Sakla, Jayaraman J. Thiagarajan

Abstract

Deep neural networks often struggle with "real-world" distribution shifts, such as environmental corruptions and out-of-distribution (OOD) anomalies. While Anchoring—a technique where models are trained on [reference, residual] pairs—was proposed to mitigate this brittleness, it has historically failed to scale with diverse reference sets. This paper, authored by researchers from Lawrence Livermore National Laboratory and Amazon, identifies the "Diversity Paradox" as the cause of this failure and introduces Reference Masking Regularization to overcome it. The resulting models demonstrate significant improvements in robustness, calibration, and safety across various architectures, including CNNs and Vision Transformers.

Introduction

Despite their success in controlled environments, deep neural networks remain notoriously brittle. A model trained on clean, high-resolution images may fail when encountering fog, digital noise, or artistic renditions of the same objects. To address this, researchers have explored Anchoring. Instead of processing a raw image $x$, an anchored model receives a pair: a reference image $\bar{r}$ and the residual $d = x - \bar{r}$. By learning the relationship between a known anchor and the input, the model is theoretically better equipped to handle variations.

However, previous implementations of anchoring faced a significant hurdle known as the Diversity Paradox. While intuition suggests that a more diverse set of anchors should improve generalization, empirical evidence showed the opposite: as the diversity of reference images increased, model performance often degraded. The authors identify that models learn to take a "shortcut" by ignoring the anchor entirely and focusing solely on the residual, thereby defeating the purpose of the anchoring mechanism.

Reference Masking Regularization

To prevent shortcut learning, the authors propose Reference Masking Regularization. This training protocol forces the model to rely on the relationship between the anchor and the residual through a simple yet effective penalty system:

  1. Masking: During training, the reference anchor is occasionally replaced with a zero vector (masked).
  2. Entropy Penalty: When the anchor is masked, the model is penalized via cross-entropy loss if it produces a confident prediction. Instead, it is forced to output a uniform distribution.
  3. Incentivizing Dependency: This logic dictates that if the model lacks an anchor, it is not allowed to be certain about the label. This effectively punishes the model for trying to classify based on the residual alone, forcing it to utilize the anchor-residual relationship.

This approach is architecture-agnostic. It functions as a training wrapper that requires only a minor modification: doubling the input channels at the first layer to accommodate the [anchor, residual] tuple. It has been successfully validated on ResNets, Swin Transformers, and standard Vision Transformers (ViTs).

Experimental Results and Insights

Robustness and Generalization

The proposed method was tested against standard benchmarks including CIFAR-10/100 and ImageNet-1K, along with their robustness variants (CIFAR-10C, ImageNet-C, ImageNet-R, and ImageNet-S).

  • CIFAR Results: The method improved corruption accuracy by 1.5% to 8.5% over standard training baselines.
  • Large-Scale Scaling: When applied to a Swin-B transformer on ImageNet-C, the method provided a +3.32% boost in accuracy on the most severe corruption levels.
  • Synergy: Anchoring was found to be synergistic with existing data augmentation techniques like Mixup and PixMix, achieving higher robustness when combined than any single technique achieved in isolation.

Safety and Calibration

Beyond accuracy, the researchers evaluated model "safety"—the ability of a model to recognize when it is viewing an anomaly. On the NINCO anomaly detection benchmark, a ViT-B16 trained with reference masking saw an AUROC increase of +4.34%. Furthermore, the models exhibited lower Expected Calibration Error (ECE), meaning their probability estimates were more reliable and less prone to overconfidence, a critical requirement for high-stakes applications like autonomous driving.

Optimization Landscape

To understand why the method works, the authors utilized 2D loss landscape visualizations. Their analysis revealed that reference masking leads the optimization process toward wider and flatter optima in the weight space. In deep learning theory, flatter optima are strongly correlated with superior generalization and stability under distribution shifts.

Conclusion

By solving the Diversity Paradox through Reference Masking Regularization, this research transforms anchoring from a niche technique into a scalable, robust training strategy. One of the most practical findings is that, unlike previous anchoring methods that required complex inference protocols, a model trained with this regularizer requires only a single random anchor at test time to achieve peak performance.

Future work may involve integrating anchoring with "model souping" (averaging weights of multiple fine-tuned models) or providing a more rigorous theoretical framework for the emergence of flatter optima through the lens of Neural Tangent Kernels.

Summary updated

Abstract

Anchoring is a recent, architecture-agnostic principle for training deep neural networks that has been shown to significantly improve uncertainty estimation, calibration, and extrapolation capabilities. In this paper, we systematically explore anchoring as a general protocol for training vision models, providing fundamental insights into its training and inference processes and their implications for generalization and safety. Despite its promise, we identify a critical problem in anchored training that can lead to an increased risk of learning undesirable shortcuts, thereby limiting its generalization capabilities. To address this, we introduce a new anchored training protocol that employs a simple regularizer to mitigate this issue and significantly enhances generalization. We empirically evaluate our proposed approach across datasets and architectures of varying scales and complexities, demonstrating substantial performance gains in generalization and safety metrics compared to the standard training protocol. The open-source code is available at https://software.llnl.gov/anchoring.
BibTeX
@inproceedings{narayanaswamy2024use,
  author = {Vivek Narayanaswamy and Kowshik Thopalli and Rushil Anirudh and Yamen Mubarka and Wesam A. Sakla and Jayaraman J. Thiagarajan},
  title = {{On the Use of Anchoring for Training Vision Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=xymhWyiZOp},
}
Figure