Distance-Based Regularisation of Deep Networks for Fine-Tuning

Henry Gouk, Timothy M. Hospedales, Massimiliano Pontil

Introduction

Fine-tuning—taking a model pre-trained on a massive dataset (like ImageNet) and adapting it to a smaller, specific task—is the backbone of modern computer vision. However, the process is often messy, relying on heuristics and trial-and-error.

This paper moves fine-tuning from an "ad-hoc art" to a principled science by proving that constraining how far weights move from their pre-trained starting point directly improves generalization. The authors introduce MARS-PGM, a simple algorithm that enforces hard constraints on weight distances using the Maximum Absolute Row Sum (MARS) norm, consistently outperforming standard penalty-based methods across various computer vision tasks.

The Problem with Current Fine-Tuning

While researchers intuitively know that keeping fine-tuned weights "close" to the original pre-trained weights helps prevent catastrophic forgetting and overfitting, existing methods (like $\ell^2$-SP) usually just add a soft Euclidean penalty to the loss function. This paper addresses three critical gaps in this standard approach:

  1. Theoretical: Why does staying "close" actually help, and how do we rigorously measure it?
  2. Metric: Is Euclidean distance the right way to measure "closeness" in a complex neural network?
  3. Optimization: Is a "soft" penalty (like weight decay) as effective as a "hard" constraint (forcing the weights to stay within a specific, rigid radius)?

Methods: Theory and Algorithm Design

The authors approach the problem through both rigorous mathematical theory and practical algorithm design:

New Generalization Bounds

They derive new bounds on the Rademacher complexity of neural networks. Crucially, these bounds depend on the distance weights travel from their initialization, rather than the total number of weights. This makes the theory much more relevant for large convolutional networks where parameter counts are massive.

The MARS Norm

The authors argue that the Maximum Absolute Row Sum (MARS) norm is a superior metric for convolutional layers compared to the standard Frobenius (Euclidean) norm. Theoretically, MARS-based bounds do not scale with the resolution of feature maps, making them significantly more stable and representative of actual network behavior.

Projected Gradient Method (PGM)

Instead of adding a penalty term to the loss function—which the optimizer can sometimes overpower or ignore—they use hard constraints. After every gradient update, the algorithm "projects" the weights back onto a sphere of radius $\gamma$ centered at the initial pre-trained weights. If the weights try to move too far, they are strictly pushed back to the boundary.

Histogram of layer-wise MARS distances for MARS-PGM, illustrating the hard constraint (dashed line) enforced by the projection step.

Key Insights and Results

The authors evaluated their approach using ResNet-101 and EfficientNetB0 architectures across seven diverse datasets, including fine-grained classification (Aircraft, Birds), textures (DTD), and general objects (Caltech-256).

  • Hard Constraints > Soft Penalties: The most striking finding is that enforcing a distance limit via projections (PGM) works significantly better than simply penalizing distance in the loss function. Penalties are "leaky" and do not strictly limit the search space in the same way.
  • Superior Accuracy: MARS-PGM achieved the best average rank across all tested datasets. On EfficientNetB0, it secured the #1 spot on every single benchmark.
  • Layer-wise Flexibility: By applying constraints on a per-layer basis, the method allows different parts of the network to adapt at different rates. For example, one might allow the final classification layer to move more than the early feature-extraction layers.
  • Robustness to Hyperparameters: The projection-based methods were far less sensitive to hyperparameter swings compared to penalty-based methods like DELTA or $\ell^2$-SP.
Performance of different regularization methods across hyperparameter values on EfficientNetB0, demonstrating the superior robustness of projection-based methods (PGM) over penalty-based ones (SP).

Limitations and Future Work

While highly effective, the method is not without limitations. It still requires tuning the radius $\gamma$ for each layer (or using a global scaling factor), which adds a step to the training pipeline. Furthermore, although the new theoretical bounds are much tighter than previous ones, they are still "vacuous" in an absolute sense (the theoretical error bound is often > 1), which remains a common issue in deep learning theory.

Finally, the paper focuses primarily on Convolutional Neural Networks (CNNs). While the underlying logic should hold, the specific effectiveness of the MARS norm for Transformer-based architectures remains an open and exciting question for future exploration.

Summary updated

Abstract

We investigate approaches to regularisation during fine-tuning of deep neural networks. First we provide a neural network generalisation bound based on Rademacher complexity that uses the distance the weights have moved from their initial values. This bound has no direct dependence on the number of weights and compares favourably to other bounds when applied to convolutional networks. Our bound is highly relevant for fine-tuning, because providing a network with a good initialisation based on transfer learning means that learning can modify the weights less, and hence achieve tighter generalisation. Inspired by this, we develop a simple yet effective fine-tuning algorithm that constrains the hypothesis class to a small sphere centred on the initial pre-trained weights, thus obtaining provably better generalisation performance than conventional transfer learning. Empirical evaluation shows that our algorithm works well, corroborating our theoretical results. It outperforms both state of the art fine-tuning competitors, and penalty-based alternatives that we show do not directly constrain the radius of the search space.
BibTeX
@inproceedings{gouk2021distance,
  author = {Henry Gouk and Timothy M. Hospedales and Massimiliano Pontil},
  title = {{Distance-Based Regularisation of Deep Networks for Fine-Tuning}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=LI3xJ1YXcZmX},
}
Figure