Calibrated Target Noise Recovers Curvature from the Gradients

Arash Jamshidi, Katsiaryna Haitsiukevich, Aristides Gionis, Kai PuolamäkiarXiv 2026 · arXiv

TL;DR

Researchers have developed a clever way to "trick" standard neural network gradients into revealing the model's underlying curvature. By injecting carefully calibrated noise into training targets, they can recover the Generalised Gauss-Newton (GGN) matrix using only aggregate mini-batch gradients—no expensive per-sample gradients or second-order derivatives required.

What Problem Does This Paper Address?

Modern machine learning models are almost exclusively trained using first-order optimization (gradients). However, their behavior is deeply governed by second-order structure, or curvature. Knowing the curvature helps in choosing better step directions (preconditioning), estimating uncertainty, and determining the optimal time to stop training.

The catch is that curvature is incredibly expensive to compute. Exact Hessians are too large to store for modern networks, and even practical surrogates like the GGN matrix typically require per-sample gradients or Jacobians. Most deep learning frameworks and distributed training systems are optimized to provide only aggregate mini-batch gradients, making these curvature-aware methods impractical at scale. This paper aims to bridge that gap: how can we extract rich geometric information from the simple batch gradients we already have?

Methods

The authors introduce the Calibrated Noisy Gradient Covariance (CNGC) estimator. The core idea is to perturb the training targets ($y$) with zero-mean noise before computing the gradient. Crucially, this noise is calibrated: its variance is scaled according to the local output curvature of the loss function and the batch size.

The paper proposes two main variants:

  1. Centred Estimator: This version computes both a "clean" gradient and a "noisy" gradient for the same batch. By subtracting the clean gradient from the noisy one, it creates an unbiased estimator of the batch GGN.
  2. Swamping Estimator: This version only uses noisy gradients, centering them around a running mean. While it introduces a small bias, it is even more lightweight and the bias vanishes as the batch size or noise level increases.

A key theoretical contribution is the proof that scaled Rademacher noise (randomly flipping between $\pm\sqrt{v_i}$) is optimal. It minimizes the estimator's variance compared to standard Gaussian noise, leading to more stable curvature recovery.

Datasets

The method was evaluated across a variety of architectures (MLPs, CNNs, Transformers) and tasks:

  • Classification: MAGIC Gamma Telescope, Bank Marketing, Adult Income, and Occupancy Detection datasets.
  • Regression: Air Quality dataset.
  • Synthetic Data: Used for controlled experiments to verify exact curvature recovery against ground-truth Hessians.

Key Insights and Ideas

  • Noise as a Probing Tool: Instead of viewing noise solely as a regularizer, this work treats it as a measurement device. By observing how the gradient "wiggles" in response to specific target perturbations, we can deduce the shape of the surrounding loss landscape.
  • Rademacher > Gaussian: While Gaussian noise is the default for many researchers, the authors show that discrete, symmetric two-point noise is superior for curvature estimation because it has a smaller fourth moment for a fixed variance.
  • Framework Compatibility: Because CNGC only requires standard batch gradients, it is "plug-and-play" for existing distributed training pipelines where per-sample operations are often unsupported or prohibitively slow.

Key Results

  • Accurate Curvature Recovery: CNGC successfully recovers the GGN matrix and matches the eigenvalue spectrum of the true Hessian, especially when using the optimal Rademacher noise. Relative operator norm between GGN and its estimates showing the effects of centring, swamping and noise distribution
  • Faster Optimization: When used as a block-diagonal preconditioner, CNGC-based methods reach target loss levels significantly faster than standard Gradient Descent and perform competitively with state-of-the-art optimizers like Adam and Sophia. Effect of CNGC as a preconditioner across various tasks
  • Principled Early Stopping: The estimated curvature provides a robust signal for early stopping. The authors demonstrate that their GGN-based criterion avoids the premature stopping issues seen in simpler gradient-based heuristics. Effect of CNGC for early stopping
Relative operator norm between GGN and its estimates showing the effects of centring, swamping and noise distribution
Effect of CNGC as a preconditioner across various tasks
Effect of CNGC for early stopping

Limitations and Future Work

  • GGN vs. Full Hessian: The method primarily recovers the GGN matrix. While a powerful surrogate, it only matches the full Hessian in specific cases like Generalized Linear Models or within the diagonal blocks of ReLU networks.
  • Computational Overhead: The "Centred" estimator requires an additional backward pass with clean targets, which increases training time per step, though this is often offset by faster overall convergence.
  • Per-example Access: The method still assumes access to individual model outputs and targets within a batch to calibrate the noise, which might require minor adjustments in some highly specialized data pipelines.
  • Scalability: For very large models, the authors still rely on diagonal or block-diagonal approximations to keep matrix inversion and storage tractable. Future work could explore more sophisticated structural approximations.

Summary updated

Abstract

Estimating curvature information from gradients alone is a fundamental problem in machine learning. The challenge becomes particularly acute in settings where only aggregate mini-batch gradients are available, yet curvature information is still required for applications such as preconditioning in optimisation, sampling, and early stopping. Perhaps surprisingly, we show that a simple target-perturbation scheme suffices to recover such curvature information from gradients alone. Our method injects zero-mean noise into the targets, with variance calibrated to the local output curvature. The covariance of the resulting noisy batch gradients then recovers practical curvature matrices, without requiring per-sample gradients, second-order derivatives, or other internal network quantities. For a broad class of non-linear models, our scheme recovers the Generalised Gauss--Newton (GGN) matrix; for feed-forward ReLU networks, it additionally recovers the diagonal blocks of the population Hessian. We demonstrate that the resulting estimator is simple, lightweight, provably optimal, and easy to integrate into existing training pipelines. Experiments demonstrate accurate curvature recovery, faster optimisation when the estimator is used for preconditioning, and the utility of the estimated GGN for early stopping.
BibTeX
@misc{jamshidi2026calibrated,
  author = {Arash Jamshidi and Katsiaryna Haitsiukevich and Aristides Gionis and Kai Puolamäki},
  title = {{Calibrated Target Noise Recovers Curvature from the Gradients}},
  year = {2026},
  eprint = {2611.07671},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07671},
}
Figure