Sobolev Training for Neural Networks

Wojciech Marian Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, Razvan Pascanu

Abstract

At the heart of deep learning we aim to use neural networks as function approximators - training them to produce outputs from inputs in emulation of a ground truth function or data creation process. In many cases we only have access to input-output pairs from the ground truth, however it is becoming more common to have access to derivatives of the target output with respect to the input - for example when the ground truth function is itself a neural network such as in network compression or distillation. Generally these target derivatives are not computed, or are ignored. This paper introduces Sobolev Training for neural networks, which is a method for incorporating these target derivatives in addition the to target values while training. By optimising neural networks to not only approximate the function's outputs but also the function's derivatives we encode additional information about the target function within the parameters of the neural network. Thereby we can improve the quality of our predictors, as well as the data-efficiency and generalization capabilities of our learned function approximation. We provide theoretical justifications for such an approach as well as examples of empirical evidence on three distinct domains: regression on classical optimisation datasets, distilling policies of an agent playing Atari, and on large-scale applications of synthetic gradients. In all three domains the use of Sobolev Training, employing target derivatives in addition to target values, results in models with higher accuracy and stronger generalisation.

Introduction

Standard supervised learning typically focuses on minimizing the error between a neural network's predictions and ground truth values. However, many modern machine learning scenarios—such as model distillation, network compression, and synthetic gradients—provide access to the derivatives of the target function with respect to its inputs. Historically, this rich geometric information has been ignored due to computational complexity.

This paper introduces Sobolev Training (ST), a method that forces networks to match both target outputs and their derivatives. By encoding this additional information, models achieve higher accuracy and significantly improved data efficiency, particularly in low-data regimes.

Methodology

Sobolev Training is named after Sobolev spaces, where distances are defined by both function values and their derivatives. The core innovation lies in modifying the loss function to include a penalty for the discrepancy between the network's input-gradient and the target function's gradient.

To address the computational burden of calculating full Jacobian matrices for high-dimensional data, the authors introduce a stochastic approximation. Instead of matching the entire matrix, the network matches 1D projections of the Jacobians onto random vectors sampled from a unit sphere. This approach makes Sobolev Training scalable and compatible with standard auto-differentiation frameworks.

Computational graph for Sobolev Training showing (a) full derivative matching and (b) stochastic approximation via random projections.

Experimental Evaluation

The effectiveness of Sobolev Training was evaluated across three distinct domains:

  1. Optimization Benchmarks: Low-dimensional regression tasks (e.g., Ackley, Rosenbrock) tested in extremely low-data settings (20 to 100 samples).
  2. Policy Distillation: Distilling an A3C agent's policy for Atari games (Pong, Breakout, Space Invaders) using 100K recorded states.
  3. Large-Scale Synthetic Gradients: Training decoupled ResNet50 architectures on CIFAR-10 and ImageNet.

Results and Discussion

Sobolev Training demonstrates a massive reduction in sample complexity. By matching derivatives, the network receives multiple constraints per data point, effectively learning the "shape" of the function with a fraction of the usual data. Unlike standard regularization techniques that enforce smoothness priors, ST matches the true complexity of the target function.

Performance comparison on optimization benchmarks. The left panel visualizes 1D function approximation, while the right panel shows the Log Test MSE across various functions and data regimes.

Key results include:

  • Regression: On the Styblinski-Tang function, ST captured the functional shape with only 20 samples, where standard training failed.
  • Distillation: ST achieved lower KL divergence and action prediction error in Atari games with minimal overfitting.
  • Scalability: The authors achieved 87.4% Top-5 accuracy on ImageNet using a decoupled ResNet50 with Sobolev synthetic gradients, outperforming standard critic-based methods.

Conclusion

While Sobolev Training can struggle with very high-frequency signals in extremely sparse data regimes, it represents a significant step forward in data-efficient deep learning. Future work may explore second-order derivatives (Hessians) to model uncertainty or applying ST to internal network activations to further enhance representation learning.

Summary updated

Abstract

At the heart of deep learning we aim to use neural networks as function approximators - training them to produce outputs from inputs in emulation of a ground truth function or data creation process. In many cases we only have access to input-output pairs from the ground truth, however it is becoming more common to have access to derivatives of the target output with respect to the input - for example when the ground truth function is itself a neural network such as in network compression or distillation. Generally these target derivatives are not computed, or are ignored. This paper introduces Sobolev Training for neural networks, which is a method for incorporating these target derivatives in addition the to target values while training. By optimising neural networks to not only approximate the function's outputs but also the function's derivatives we encode additional information about the target function within the parameters of the neural network. Thereby we can improve the quality of our predictors, as well as the data-efficiency and generalization capabilities of our learned function approximation. We provide theoretical justifications for such an approach as well as examples of empirical evidence on three distinct domains: regression on classical optimisation datasets, distilling policies of an agent playing Atari, and on large-scale applications of synthetic gradients. In all three domains the use of Sobolev Training, employing target derivatives in addition to target values, results in models with higher accuracy and stronger generalisation.
BibTeX
@inproceedings{czarnecki2017sobolev,
  author = {Wojciech Marian Czarnecki and Simon Osindero and Max Jaderberg and Grzegorz Swirszcz and Razvan Pascanu},
  title = {{Sobolev Training for Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=ry-PNP-OZr},
}
Figure