Many scientific neural networks are useful because of their derivatives, not their outputs. A network that represents a spacetime metric is queried for curvature, which comes from second derivatives of its output. A network that stands in for a power-grid optimizer is used for its sensitivities: how the predicted solution moves when the inputs change. A network for a material model is used for stress, which is the derivative of a predicted energy. These networks are often trained with Sobolev training, meaning the loss penalizes errors in both the output and its input derivatives. The next step is to deploy them on embedded controllers, or to store them more cheaply. That requires weight compression: quantization (storing weights in few bits) or pruning (removing weights). Every standard compression method decides how to round or prune by checking only how well each compressed layer reproduces the original layer's outputs on a small calibration set. Nothing in that procedure checks the derivatives that the network exists to supply.
This note proposes tangent calibration. It is a cheap, drop-in change to layer-wise post-training compression that keeps input derivatives intact as well as output values. The key observation is simple. Inside a linear layer $y = Wx$, every derivative of $y$ with respect to the network input is $W$ applied to the matching derivative of $x$. So derivatives are just extra "inputs" that the same weight matrix must handle correctly. If the derivative vectors computed in the dense network are added to the calibration set, existing solvers such as GPTQ and SparseGPT become derivative-aware. No double backpropagation is needed and the solver code does not change. We also test the claim that motivates the method: at a fixed bit budget, standard compression degrades a Sobolev-trained network's Hessian faster than its Jacobian, and its Jacobian faster than its values, with the gap widening for high-frequency activations.
Where this sits
Two bodies of work meet here without overlapping.
The Sobolev line shows why derivatives are the deliverable. Sobolev Training for Neural Networks introduced derivative-supervised training. It showed large sample-efficiency gains, for example 20 points beating 100 on Styblinski-Tang, and proposed matching derivatives along random directions to keep the cost down. We reuse that random-direction idea for calibration. Einstein Fields fits numerical-relativity metrics with Jacobian and Hessian supervision. It reaches up to 4,000x data compression at about $10^{-7}$ relative error, and its derivatives from automatic differentiation are up to 100,000x more accurate than finite-difference stencils on the original grid. Its weights, however, stay in float32. The paper also notes that precision already limits it, since float32 error accumulates in rollouts. Sobolev Training of End-to-End Optimization Proxies matches proxy Jacobians to sensitivities from the optimizer's optimality conditions. This cuts optimal-power-flow error by up to 56% and median worst-case constraint violation about 4x. It also found that supervising the full dense Jacobian fails, and that randomly masking 75–95% of its entries helps.
These papers motivate the proposal: they establish that derivative fidelity drives downstream validity. The masking result also constrains it. It warns that adding every derivative constraint at once can over-constrain the fit, so tangent calibration uses a few random directions rather than full Jacobians. Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives adds a cheap testbed: coordinate networks that map pixel position to color (implicit neural representations) on images and audio. It also adds a mechanistic hint. Periodic-activation networks (SIREN) carry high-order derivative signal that ReLU networks lose, which suggests derivatives are fragile in exactly the networks that need them.
The compression line supplies the methods we modify. LLMCBench benchmarks GPTQ, AWQ, SparseGPT, Wanda and others. It finds that different compression types preserve different capabilities: quantization keeps factual recall, while pruning better keeps reasoning. This is evidence that output-level reconstruction does not preserve all functional properties equally. Restoring Pruned Large Language Models via Lost Component Compensation finds that small-singular-value directions of the activation gap carry signal that matters disproportionately. We hypothesize that derivative information sits in similarly under-weighted directions. Output reconstruction ignores those directions because ordinary calibration inputs rarely excite them, but tangent vectors do.
The closest prior connections are not this proposal. Czarnecki et al.'s policy distillation transfers input gradients into a smaller dense student trained from scratch. Continual Learning of Physical Systems via Derivative Distillation distills derivatives into a same-size network to prevent forgetting. Neither compresses the weights of a trained network, and neither works in the post-training, calibration-only setting.
Core Claim
Let a trained surrogate have layers $y_\ell = W_\ell x_\ell$ followed by nonlinearities. Pick random unit input directions $v$. Forward-mode differentiation through the dense network gives, at every layer, the first-order tangent $t_\ell = \partial_v x_\ell$ and the second-order tangent $s_\ell = \partial^2_v x_\ell$. Because the layer is linear, the compressed layer's derivative error equals $(\hat W_\ell - W_\ell)$ applied to these vectors. We therefore compress each layer by solving
$$ \hat W_\ell = \arg\min_{\hat W \in \mathcal{C}} \; \|(\hat W - W_\ell) X\|_F^2 + \lambda_1 \|(\hat W - W_\ell) T\|_F^2 + \lambda_2 \|(\hat W - W_\ell) S\|_F^2 , $$
where $\mathcal{C}$ is the set of allowed compressed matrices (a 3- or 4-bit grid, or a sparsity pattern). The columns of $X$, $T$, $S$ are the calibration activations, first-order tangents and second-order tangents. The weights $\lambda_1, \lambda_2$ trade off value against derivative fidelity.
This is exactly the standard layer-wise problem with the second-moment matrix $XX^\top$ replaced by
$$ H = XX^\top + \lambda_1 TT^\top + \lambda_2 SS^\top, $$
so GPTQ, SparseGPT and similar solvers run unmodified on $H$. The extra cost is a few forward-mode passes over the calibration set.
The method has a known gap. Compressing earlier layers changes later layers' activations, and through the curvature of the nonlinearity this corrupts derivatives in ways the per-layer objective does not see. We measure that residual. Where it dominates, we add a short end-to-end Sobolev fine-tune of the quantization scales only.
The accompanying hypothesis can be checked against first-order perturbation theory. For a SIREN layer $\sin(\omega_0 W x)$, a weight perturbation enters the $k$-th derivative multiplied by roughly $\omega_0^k$ times the tangent norms, while it enters the value only at order one. Relative error under output-only calibration should therefore grow with derivative order and with the frequency $\omega_0$. Tangent calibration should flatten that growth.
Research Program
The work has three parts.
First, a diagnostic study. Take published-style Sobolev-trained surrogates in three settings: Einstein Fields on Schwarzschild and Kerr metrics and the Einstein Toolkit neutron star; optimal-power-flow proxies on PGLib IEEE-300 and PEGASE-1k; and SIREN representations of Set5/DIV2K images and Bach cello audio. Compress each with standard GPTQ and SparseGPT across 8, 6, 4, 3 and 2 bits and 30–70% sparsity. For each setting, plot four error-versus-size curves: values, Jacobian, Hessian, and a downstream quantity. The downstream quantity is the Kretschmann curvature scalar for the metrics, constraint violation and sensitivity error for the power-flow proxies, and gradient-image error for the images. This answers the question the file marks as entirely open: whether weight compression destroys derivatives before values, and at what bit-width the cliff appears for each derivative order.
Second, the method. Build tangent calibration on top of GPTQ and SparseGPT. The design choices to study are the number of random directions, an optional mask on tangent coordinates (following the masking result of the optimization-proxy paper), and the order of tangents included.
Third, the analysis. Compare the spectrum of $H$ with that of $XX^\top$, to test whether tangents excite directions that values leave unexcited. This is our analogue of RestoreLCC's minor-component finding. Also check whether the predicted $\omega_0^k$ scaling appears empirically, across SIREN networks retrained at several frequencies.
Baselines
The naive reference is the uncompressed float32 Sobolev-trained surrogate, together with plain round-to-nearest quantization and magnitude pruning at each budget. These fix the ceiling and the floor. For the ceiling, the file reports up to 4,000x data compression at about $10^{-7}$ relative error on analytic black holes for Einstein Fields Einstein Fields. It also reports PSNR 30.28 on Set5/DIV2K and 41.23 on Bach audio for Sobolev-trained SIREN Sobolev Training for INRs, and up to 56% MSE reduction with a 4x median violation reduction for Sobolev optimal-power-flow proxies Sobolev Training of End-to-End Optimization Proxies. All of these are uncompressed-weight numbers. No reported result was found on Lacuna for weight-quantized or pruned Sobolev-trained surrogates, so we must produce every compressed baseline ourselves.
The strongest existing method is output-calibrated GPTQ for quantization and SparseGPT for pruning, the solvers benchmarked in LLMCBench. They are the right competitor because tangent calibration differs from them only in the calibration matrix. They will use the identical calibration points, group sizes, damping and solver. To match information content rather than just point count, they will also get a calibration set enlarged by the number of tangent columns we add. Beating them means lower Jacobian and Hessian error at equal bits, with value error within 10% of theirs. More decisively, it means reaching the uncompressed surrogate's downstream metric (curvature error, constraint violation) at least one bit lower.
The key ablation isolates the derivative signal from mere extra data. It replaces the tangent columns with activations at finitely perturbed inputs $x + \epsilon v$, equal in number, across a sweep of $\epsilon$. A second ablation drops the second-order tangents to separate Jacobian from Hessian contributions.
The expensive upper reference is end-to-end Sobolev quantization-aware training. It fine-tunes the compressed network with the full value-plus-derivative loss using double backpropagation, and we give it ten times our compute. We report how much of its gain tangent calibration recovers per GPU-hour.
Why not just…?
Why not just calibrate on more, densely spaced points instead of computing tangents? This is the most serious alternative. In the limit it is nearly equivalent: the difference between activations at $x+\epsilon v$ and at $x$, divided by $\epsilon$, approaches $t_\ell$. The problem is weighting. In $XX^\top$ that difference is an $O(\epsilon)$ perturbation of nearly duplicate columns, so the solver effectively ignores it. Recovering it requires rescaling by $1/\epsilon$, which reintroduces the finite-difference noise that Einstein Fields reports as orders of magnitude worse than automatic differentiation. Sobolev training's own evidence also points this way: 20 derivative-supervised points beat 100 plain ones Sobolev Training for Neural Networks. Still, the perturbed-point ablation will decide the question. If some $\epsilon$ matches tangent calibration on Hessian error, we will report the method as "a principled choice of calibration geometry" rather than as a new signal.
Why not just run Sobolev fine-tuning after ordinary compression? It probably works best, and we include it as an upper reference. It does, however, need double backpropagation, which the file identifies as a major memory and time cost. The full-Jacobian version is also unstable Sobolev Training of End-to-End Optimization Proxies. The claim we test is narrower: post-training tangent calibration closes most of the derivative gap at a small fraction of the cost. It can also initialize fine-tuning, which should then converge faster. Both claims are measured directly.
Why not just train a smaller dense Sobolev network, or keep 8-bit weights? A smaller network at equal storage is a real competitor, and it is what Czarnecki et al.'s distillation implicitly does. We compare at matched bytes: a dense float32 network with roughly a quarter of the width against a 4-bit compressed full-width network. If the small dense network wins on curvature error, that is a negative result about weight compression for these surrogates, and worth publishing. On 8-bit weights: LLMCBench's finding of over 95% retention concerns language-model accuracy at moderate bit-widths, not derivatives. Einstein Fields already strains float32 precision, so it is not obvious that 8 bits is "good enough" for Hessians. The diagnostic curves answer this directly.
Decisive Checks
The proposal succeeds if three things hold. First, the diagnostic study shows that output-calibrated GPTQ loses Hessian accuracy at a higher bit-width than it loses value accuracy, on at least two of the three domains, with the gap growing with SIREN frequency. Second, tangent calibration shifts that Hessian cliff down by at least one bit, or allows about 10 percentage points more sparsity at matched curvature error. It must do this without losing to the perturbed-point ablation. Third, on the optimal-power-flow proxies, the compressed network keeps the median constraint-violation advantage that Sobolev training buys over value-only training. That last check is the one practitioners care about: a 4-bit surrogate that still behaves like a Sobolev-trained one inside the solver it serves.
References
- Sobolev Training for Neural Networks
- Einstein Fields: A Neural Perspective To Computational General Relativity
- Sobolev Training of End-to-End Optimization Proxies
- Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives
- LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment
- Restoring Pruned Large Language Models via Lost Component Compensation
- Continual Learning of Physical Systems via Derivative Distillation