On the Use of Anchoring for Training Vision Models

Vivek Sivaraman Narayanaswamy, Kowshik Thopalli, Rushil Anirudh, Yamen Mubarka, Wesam Sakla, Jayaraman J. Thiagarajan

TL;DR

This paper explores "anchoring," a novel training technique for deep neural networks that improves generalization and safety. They identify a key limitation where anchoring can lead to "shortcuts," hindering generalization, and propose a simple "reference masking" regularization to fix this, achieving significant performance gains across various vision models and datasets.

What Problem Does This Paper Address?

Deep neural networks, while powerful, often struggle with generalization to unseen data (out-of-distribution generalization), calibration (overconfidence in predictions), and extrapolation. Anchoring, a recently proposed architecture-agnostic training principle, aims to address these issues by reparameterizing inputs. However, the authors found that existing anchoring methods don't fully leverage the diversity of references, leading to a risk of learning undesirable shortcuts and limiting generalization. This paper investigates this limitation and proposes a solution to unlock anchoring's full potential.

Methods

The core of the paper revolves around the "anchoring" technique. Instead of feeding an input x directly to a network F, anchoring reparameterizes it into a tuple [r, d], where r is a reference sample and d = x - r is the residual. The network then learns to predict the target y from this [r, d] tuple. This forces the model to learn a mapping from the joint space of references and residuals.

The paper's main contribution is a new anchored training protocol that introduces a reference masking regularization. This regularization randomly "masks" the reference part of the input tuple [r, x-r] during training, effectively feeding [0, x-r] to the network. To prevent the model from exploiting this shortcut, a cross-entropy loss is added to penalize meaningful predictions when the reference is masked, encouraging the model to rely on both reference and residual.

Datasets

The authors evaluated their approach on several standard vision datasets:

  • CIFAR-10 and CIFAR-100: Used for training and evaluating CNNs (ResNet-18, WideResNet40-2).
  • ImageNet-1K: A large-scale benchmark used for training and evaluating vision transformers (SWINv2-T, SWINv2-S, SWINv2-B, ViT-B-16).

They also used various out-of-distribution (OOD) datasets for evaluation, including:

  • CIFAR10C/C, CIFAR100C/C: Corrupted versions of CIFAR datasets.
  • ImageNet-C, ImageNet-R, ImageNet-S: Various benchmarks for evaluating robustness to natural corruptions, renditions, and sketches.
  • Vision OOD datasets (iSUN, Textures, Places365) and NINCO: Used for anomaly rejection evaluation.

Key Insights and Ideas

  • Anchoring's promise and peril: Anchoring shows significant potential for improving generalization, calibration, and extrapolation, but it's not a silver bullet.
  • The "shortcut" problem: The core limitation identified is that anchored training can lead to models learning to rely too heavily on the residual d and ignore the reference r, especially with diverse reference sets. This hinders generalization.
  • Reference diversity is key, but not fully leveraged: While diverse references are theoretically beneficial, the standard anchoring protocol doesn't effectively utilize this diversity.
  • Inference protocols don't fix the training issue: Sophisticated inference strategies (like marginalization or bilinear transduction) don't fundamentally solve the generalization problem caused by the training limitation.
  • Reference masking as a simple fix: The proposed reference masking regularization is a simple yet effective way to force the model to learn from both reference and residual, thereby improving generalization.
  • Anchoring as a generic wrapper: The technique is architecture-agnostic and can be combined with existing training strategies like data augmentations and optimizers.

Key Results

  • Significant generalization gains: The proposed reference masking approach consistently outperformed standard training and vanilla anchoring across various datasets and architectures, showing improvements of 1.54% – 8.54% on average for corruption accuracies.
  • Improved robustness: The method demonstrated better robustness to label noise, with improvements of up to 4% at 20% label noise compared to standard training.
  • Enhanced anomaly rejection and calibration: Anchoring, especially with the proposed regularization, led to better anomaly detection and lower calibration errors, making models more reliable.
  • Wider and flatter optima: Analysis of the accuracy landscape suggested that the proposed approach finds wider and flatter optima, which are known to correlate with better generalization.
  • No significant inference overhead: The proposed method adds minimal to no extra computational cost during inference.

Limitations and Future Work

  • Theoretical understanding: The paper acknowledges that a theoretical understanding of why anchoring works and its generalization properties is still lacking and suggests this as an important future direction.
  • Integration with state-of-the-art: The authors suggest integrating anchoring with techniques like model souping or fine-tuning large pre-trained models to potentially achieve even better results.
  • Hyperparameter tuning: While they found good values for a (the masking probability), further exploration of hyperparameter sensitivity could be beneficial.

Summary updated

Abstract

Anchoring is a recent, architecture-agnostic principle for training deep neural networks that has been shown to significantly improve uncertainty estimation, calibration, and extrapolation capabilities. In this paper, we systematically explore anchoring as a general protocol for training vision models, providing fundamental insights into its training and inference processes and their implications for generalization and safety. Despite its promise, we identify a critical problem in anchored training that can lead to an increased risk of learning undesirable shortcuts, thereby limiting its generalization capabilities. To address this, we introduce a new anchored training protocol that employs a simple regularizer to mitigate this issue and significantly enhances generalization. We empirically evaluate our proposed approach across datasets and architectures of varying scales and complexities, demonstrating substantial performance gains in generalization and safety metrics compared to the standard training protocol.
BibTeX
@inproceedings{narayanaswamy2024use,
  author = {Vivek Sivaraman Narayanaswamy and Kowshik Thopalli and Rushil Anirudh and Yamen Mubarka and Wesam Sakla and Jayaraman J. Thiagarajan},
  title = {{On the Use of Anchoring for Training Vision Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=xymhWyiZOp},
}
Figure