Abstract
At the heart of deep learning we aim to use neural networks as function approximators - training them to produce outputs from inputs in emulation of a ground truth function or data creation process. In many cases we only have access to input-output pairs from the ground truth, however it is becoming more common to have access to derivatives of the target output with respect to the input - for example when the ground truth function is itself a neural network such as in network compression or distillation. Generally these target derivatives are not computed, or are ignored. This paper introduces Sobolev Training for neural networks, which is a method for incorporating these target derivatives in addition the to target values while training. By optimising neural networks to not only approximate the function's outputs but also the function's derivatives we encode additional information about the target function within the parameters of the neural network. Thereby we can improve the quality of our predictors, as well as the data-efficiency and generalization capabilities of our learned function approximation. We provide theoretical justifications for such an approach as well as examples of empirical evidence on three distinct domains: regression on classical optimisation datasets, distilling policies of an agent playing Atari, and on large-scale applications of synthetic gradients. In all three domains the use of Sobolev Training, employing target derivatives in addition to target values, results in models with higher accuracy and stronger generalisation.
Introduction
Standard supervised learning typically focuses on minimizing the error between a neural network's predictions and ground truth values. However, many modern machine learning scenarios—such as model distillation, network compression, and synthetic gradients—provide access to the derivatives of the target function with respect to its inputs. Historically, this rich geometric information has been ignored due to computational complexity.
This paper introduces Sobolev Training (ST), a method that forces networks to match both target outputs and their derivatives. By encoding this additional information, models achieve higher accuracy and significantly improved data efficiency, particularly in low-data regimes.
Methodology
Sobolev Training is named after Sobolev spaces, where distances are defined by both function values and their derivatives. The core innovation lies in modifying the loss function to include a penalty for the discrepancy between the network's input-gradient and the target function's gradient.
To address the computational burden of calculating full Jacobian matrices for high-dimensional data, the authors introduce a stochastic approximation. Instead of matching the entire matrix, the network matches 1D projections of the Jacobians onto random vectors sampled from a unit sphere. This approach makes Sobolev Training scalable and compatible with standard auto-differentiation frameworks.
Experimental Evaluation
The effectiveness of Sobolev Training was evaluated across three distinct domains:
- Optimization Benchmarks: Low-dimensional regression tasks (e.g., Ackley, Rosenbrock) tested in extremely low-data settings (20 to 100 samples).
- Policy Distillation: Distilling an A3C agent's policy for Atari games (Pong, Breakout, Space Invaders) using 100K recorded states.
- Large-Scale Synthetic Gradients: Training decoupled ResNet50 architectures on CIFAR-10 and ImageNet.
Results and Discussion
Sobolev Training demonstrates a massive reduction in sample complexity. By matching derivatives, the network receives multiple constraints per data point, effectively learning the "shape" of the function with a fraction of the usual data. Unlike standard regularization techniques that enforce smoothness priors, ST matches the true complexity of the target function.
Key results include:
- Regression: On the Styblinski-Tang function, ST captured the functional shape with only 20 samples, where standard training failed.
- Distillation: ST achieved lower KL divergence and action prediction error in Atari games with minimal overfitting.
- Scalability: The authors achieved 87.4% Top-5 accuracy on ImageNet using a decoupled ResNet50 with Sobolev synthetic gradients, outperforming standard critic-based methods.
Conclusion
While Sobolev Training can struggle with very high-frequency signals in extremely sparse data regimes, it represents a significant step forward in data-efficient deep learning. Future work may explore second-order derivatives (Hessians) to model uncertainty or applying ST to internal network activations to further enhance representation learning.
Summary updated