Gradients as Features for Deep Representation Learning

Fangzhou Mu, Yingyu Liang, Yin Li

Introduction

In deep learning, representation learning is frequently used to adapt a model trained on a massive dataset (like ImageNet) to a new, specialized task. The standard approach is to treat the pre-trained network as a fixed feature extractor, pulling out the activations (the outputs of the neurons) and training a simple linear classifier on top.

While this "linear probing" method is highly efficient, it often underperforms compared to fine-tuning—the process of updating all the weights of the deep network for the new task. However, full fine-tuning is computationally expensive, prone to overfitting on small datasets, and requires managing complex hyperparameter schedules.

To address this, researchers propose a novel middle ground: using the gradients of the model parameters alongside standard activations. By combining these into a specialized linear model, the authors create a computationally efficient "local linear approximation" of full fine-tuning that significantly outperforms standard linear probes across unsupervised, self-supervised, and transfer learning tasks.

Methodology: Gradients as Features

The core innovation of this paper is a novel linear model that incorporates both activation-based features and gradient-based features.

  1. Extracting Gradients: For a given input, the method calculates the gradient of the model's parameters with respect to a task-specific loss. Intuitively, while activations tell you what the network currently sees, gradients tell you how the network would need to change to better understand that specific sample.
  2. The Linear Model: The researchers designed a classifier that takes the form of a first-order Taylor expansion. Mathematically, this approximates how the network's output would change if one were to perform a small step of fine-tuning.
Architecture combining activation and gradient features
  1. Scalable Training via JVP: Computing and storing raw gradients for every parameter in a deep network is incredibly memory-intensive. To solve this, the authors utilize Jacobian-vector products (JVP) using forward-mode automatic differentiation. This allows them to train the model without ever explicitly storing the massive gradient matrices, keeping the computational cost comparable to standard training.

Key Insights

  • The NTK Connection: The approach is heavily inspired by Neural Tangent Kernel (NTK) theory, which suggests that sufficiently wide neural networks behave like linear models in their gradients during training. This paper successfully materializes this theory for practical, finite-width networks.
  • Pre-training is the Secret Sauce: A critical finding is that gradients from a randomly initialized network are virtually useless. The discriminative power of gradient features is a direct consequence of the pre-training process.
  • Top-Heavy Gradients: Ablation studies showed that gradients from the topmost layers of the network provide the most significant performance gains. Adding gradients from deeper, earlier layers offers diminishing returns and increases noise.
  • Bridging the Gap: The model effectively acts as a "local" version of fine-tuning. If the new task is semantically close to the pre-training task, this linear approximation is incredibly accurate.

Experimental Results

The method was evaluated across diverse settings, including unsupervised learning (BiGAN, VAE), self-supervised learning (Jigsaw, Colorization), and transfer learning (ResNet on ImageNet). Target datasets included SVHN, CIFAR-10, CIFAR-100, PASCAL VOC07, and MS COCO.

In almost every test case, the "Full Model" (activations + gradients) outperformed the "Activation-only" baseline.

Performance comparison on SVHN, CIFAR-10, and CIFAR-100

For example, on CIFAR-10 using a BiGAN encoder, accuracy jumped from 62.87% (activations only) to 70.51% (full model). Furthermore, in transfer learning from ImageNet to VOC07, the proposed method achieved 84.95 mAP, slightly outperforming full fine-tuning (84.14 mAP) in that specific setup. Because it remains a linear model at its core, it is much faster to train and iterate upon than a full deep-network fine-tuning pipeline.

Limitations and Future Work

While powerful, the authors note a few limitations. First, the quality of the approximation depends on the "semantic gap" between tasks. If the pre-training task is vastly different from the target task, the local linear approximation may be less effective. Second, choosing which layers contribute gradients is currently based on heuristics (usually the topmost layers); developing a more principled layer-selection strategy is a promising direction for future research. Finally, while highly effective for visual tasks, extending this gradient-feature approach to Natural Language Processing (NLP) or Reinforcement Learning remains an open area for exploration.

Summary updated

Abstract

Given a pre-trained model, we explored the per-sample gradients of the model parameters relative to a task-specific loss, and constructed a linear model that combines gradients of model parameters and the activation of the model.
BibTeX
@inproceedings{mu2020gradients,
  author = {Fangzhou Mu and Yingyu Liang and Yin Li},
  title = {{Gradients as Features for Deep Representation Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=1RW2VIIL4My},
}
Figure