Distance-Based Regularisation of Deep Networks for Fine-Tuning

Henry Gouk, Timothy M. Hospedales, Massimiliano Pontil

Abstract

We investigate approaches to regularisation during fine-tuning of deep neural networks. First we provide a neural network generalisation bound based on Rademacher complexity that uses the distance the weights have moved from their initial values. This bound has no direct dependence on the number of weights and compares favourably to other bounds when applied to convolutional networks. Our bound is highly relevant for fine-tuning, because providing a network with a good initialisation based on transfer learning means that learning can modify the weights less, and hence achieve tighter generalisation. Inspired by this, we develop a simple yet effective fine-tuning algorithm that constrains the hypothesis class to a small sphere centred on the initial pre-trained weights, thus obtaining provably better generalisation performance than conventional transfer learning. Empirical evaluation shows that our algorithm works well, corroborating our theoretical results. It outperforms both state of the art fine-tuning competitors, and penalty-based alternatives that we show do not directly constrain the radius of the search space.

Introduction

Fine-tuning pre-trained models is a cornerstone of modern deep learning, yet the process often relies on ad-hoc trial-and-error. Standard regularizers, such as $\ell^2$-SP, attempt to keep fine-tuned weights near their pre-trained initialization by adding a penalty term to the loss function. However, this approach has two significant flaws: the Euclidean ($\ell^2$) distance metric does not account for the specific structural properties of convolutional layers, and the use of "soft" penalties does not strictly bound the hypothesis space, especially when using adaptive optimizers like Adam.

Methodology

This research proposes a more principled framework for fine-tuning based on two key innovations: the MARS norm and the Projected Gradient Method (PGM).

The MARS Norm

The Maximum Absolute Row Sum (MARS) norm is introduced as a superior metric for Convolutional Neural Networks (CNNs). Theoretically, generalization bounds derived from the MARS norm are independent of the number of weights and feature map resolution. This makes the bounds significantly tighter and more theoretically sound for convolutional architectures than standard Frobenius or spectral norms.

Hard Constraints via PGM

Instead of soft penalties, the authors advocate for hard constraints. In the Projected Gradient Method (PGM), weights are explicitly projected back onto a sphere of radius $\gamma$ centered at the initial weights after each update. This ensures the model remains within a strictly defined hypothesis space, whereas penalty-based methods often allow the model to wander far from its starting point.

Distribution of MARS distances across layers for different regularization methods. MARS-PGM (b, f) enforces a strict hard constraint (dashed line) compared to the spread seen in penalty-based methods.

Experimental Evaluation

The proposed methods, MARS-PGM and $\ell^2$-PGM, were evaluated using ResNet-101 and EfficientNetB0 architectures on seven diverse datasets, including Aircraft, Flowers, and Caltech-256.

Key Results

Empirical results demonstrate that MARS-PGM consistently achieves superior accuracy and robustness. For instance, on the Aircraft dataset with ResNet-101, MARS-PGM reached 72.04% accuracy, significantly outperforming standard fine-tuning (51.81%) and the popular $\ell^2$-SP (60.16%).

Accuracy vs. regularization strength (c) for ResNet101 and EfficientNetB0. Projection-based methods show competitive performance and distinct behavior compared to penalty-based approaches.

Discussion and Insights

  • Capacity Control: The projection radius $\gamma$ acts as a direct knob for model capacity. A small radius prevents overfitting on tiny datasets, while a larger radius allows the model to learn more complex features when data is abundant.
  • Constraints vs. Penalties: The study highlights that in deep learning, penalties and constraints are not equivalent. Penalty-based methods often fail to restrict the search space effectively because training rarely reaches a true stationary point.
  • CNN-Friendly Bounds: The MARS norm's effectiveness stems from its treatment of weight sharing in convolutional layers, leading to complexity bounds that are orders of magnitude tighter than those found in existing literature.

Conclusion

This work provides a theoretical and empirical foundation for distance-based regularization in fine-tuning. While the combination of PGM with Adam remains a heuristic and the bounds are currently too loose for automated hyperparameter selection, the approach offers a significant step toward more robust and predictable transfer learning. Future work will investigate automated ways to allocate fine-tuning capacity across different network layers.

Summary updated

Abstract

We investigate approaches to regularisation during fine-tuning of deep neural networks. First we provide a neural network generalisation bound based on Rademacher complexity that uses the distance the weights have moved from their initial values. This bound has no direct dependence on the number of weights and compares favourably to other bounds when applied to convolutional networks. Our bound is highly relevant for fine-tuning, because providing a network with a good initialisation based on transfer learning means that learning can modify the weights less, and hence achieve tighter generalisation. Inspired by this, we develop a simple yet effective fine-tuning algorithm that constrains the hypothesis class to a small sphere centred on the initial pre-trained weights, thus obtaining provably better generalisation performance than conventional transfer learning. Empirical evaluation shows that our algorithm works well, corroborating our theoretical results. It outperforms both state of the art fine-tuning competitors, and penalty-based alternatives that we show do not directly constrain the radius of the search space.
BibTeX
@inproceedings{gouk2021distance,
  author = {Henry Gouk and Timothy M. Hospedales and Massimiliano Pontil},
  title = {{Distance-Based Regularisation of Deep Networks for Fine-Tuning}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=LI3xJ1YXcZmX},
}

Loading related content...

Figure