Many learning systems now place an optimization solver inside a neural network. A network predicts the costs of a quadratic program (an optimization problem with a quadratic objective and linear constraints), an off-the-shelf solver such as Gurobi or OSQP finds the best decision, and training needs the gradient of that decision with respect to the predicted costs. The best current methods compute this gradient solver-agnostically: they let any solver do the forward pass and then differentiate the optimality conditions at its answer. They do not trace back through the solver's internal steps. These methods quietly assume that the answer is essentially exact. They have to, because the gradient formula is only valid at a true optimum and because they must decide sharply which inequality constraints are binding. Running a solver to high precision on every training step is expensive, and work on differentiable physics shows that much of that precision is wasted early in training.
We propose to break this coupling between solver precision and gradient correctness. Solve loosely, then take a few analytic residual-correction steps that pull the loose answer onto the exact optimum of a smoothed version of the problem, and differentiate only through the last correction. These steps are the classical defect-correction idea at the heart of iterative residual learning, where one measures how far the current answer is from satisfying the equations and fixes that error. Because the correction is a Newton step on a smooth, strongly convex surrogate, the resulting gradient is exact for the surrogate. Its error is controlled by two quantities the user sets, the smoothing width and the number of corrections. The solver's stopping tolerance no longer determines gradient quality. That lets the expensive black-box solve run at tolerances orders of magnitude looser than current methods tolerate.
Core Claim
Three existing pieces each solve part of the problem. First, dXPP replaces inequality constraints in the backward pass with a softplus penalty of width $\delta$. This turns gradient computation into a symmetric positive definite primal system, with relative gradient errors of $10^{-7}$ to $10^{-4}$ against exact KKT gradients, where KKT refers to the standard optimality conditions for constrained problems. It evaluates that system at whatever point the solver returns. Second, One-step differentiation of iterative algorithms shows that differentiating only the final step of an iteration gives a Jacobian whose error is bounded by the iteration's contraction rate, and that the error nearly vanishes for Newton's method. Third, PRDP shows that unconverged solver states give usable gradients early in training.
Our claim is that these compose into something none of them provides alone. Let the smoothed problem be
$$ P_\delta(z;\theta) = \tfrac12 z^\top Q z + q^\top z + \tfrac{\rho}{2}\|Az-b\|^2 + \mu\,\delta \sum_i \mathrm{softplus}\!\left(\frac{g_i^\top z - h_i}{\delta}\right), $$
where $z$ is the decision, $\theta=(Q,q,A,b,G,h)$ are the predicted problem data, $g_i^\top z \le h_i$ are the inequality constraints, and $\rho,\mu$ are penalty weights. Starting from the solver's loose answer $z_0$, we apply $k$ Newton corrections $z_{j+1} = z_j - \nabla^2 P_\delta(z_j)^{-1}\nabla P_\delta(z_j;\theta)$. We stop gradients through $z_0,\dots,z_{k-1}$ and differentiate only the last step.
Inside Newton's quadratic-convergence region, the distance to the smoothed optimum $z^*_\delta$ shrinks roughly as $\|z_0 - z_\delta^*\|^{2^k}$. The one-step Jacobian error inherits that rate through the contraction bound of One-step differentiation. Total gradient error is then the correction error plus an $O(\delta)$ smoothing bias, which shrinks as $\delta$ does, per the convergence result in dXPP.
There is a catch, and we treat it as central. The curvature of the softplus term grows like $1/\delta$, so the Newton region shrinks with $\delta$. The solver tolerance must therefore be roughly on the order of $\delta$, not on the order of the target gradient accuracy. That is still a large saving, because useful values of $\delta$ are far looser than the $10^{-8}$-scale tolerances that active-set methods need. The coupling suggests a schedule that loosens or tightens $\delta$ and solver tolerance together.
This also addresses the main failure of active-set methods. dQP and FFOLayer must classify each constraint as binding or not. A blurry or degenerate solution breaks that classification. The smoothed correction never classifies.
Research Program
We would build the correction layer as a drop-in wrapper around the black-box interface of dQP, which already wraps more than 15 solvers. For large sparse problems, Newton systems would use the sparse Cholesky backend of the dXPP codebase. Each correction costs one factorization of the same kind of matrix that dXPP's backward pass already factors. With a fixed sparsity pattern, symbolic analysis is reused, so $k$ corrections cost roughly $k$ extra backward passes. Equality constraints enter through the quadratic penalty or, where that is too ill-conditioned, through a reduced-space elimination.
The analysis has three parts. First, we would prove a gradient-error bound of the form (correction error)$^{2^k}$ plus $O(\delta)$. The bound would state explicitly the basin condition linking solver tolerance to $\delta$, and it would hold for strongly convex $Q$. Second, we would characterize what happens at degenerate vertices, where active-set Jacobians are undefined but the smoothed gradient remains well defined. Third, we would design a PRDP-style controller that sets $(\delta, \text{tolerance}, k)$ from validation plateaus. PRDP adjusts only an iteration count, so this controller moves its idea into the optimization-layer setting.
Object Representations as Fixed Points motivates the design choice of differentiating a single step at the end. It found that a single differentiated step at the fixed point stabilized training of iterative refinement and cut CLEVR-Mirror reconstruction error from 67.04 to 10.66. It applied this to heuristic attention updates. We apply it to a step whose fixed point is a certified optimum.
The residual-learning literature mainly constrains the design. Staged correctors fail when the base estimate is poor, as reported in Utility Decomposition with Deep Corrections. Unconstrained residual additions break constraints. Both concerns argue for an analytic corrector whose residual target is defined by the problem itself, the gradient of $P_\delta$, rather than a learned one. HyPINO offers the counterexample worth testing. It feeds the residual of a PDE back through a learned hypernetwork for up to 10 rounds and cuts error by over 100x, which suggests learned correctors can work well. We keep a learned corrector as an ablation, not the method.
Baselines
The naive reference point is the current practice: run the solver to tight tolerance, around $10^{-9}$, and differentiate with dQP or dXPP. It defines the gradient we treat as ground truth and the wall-clock cost we aim to beat. dQP is reported to solve 100% of the 129 Maros-Meszaros problems, while OptNet and SCQPTH fail on more than half. On the multi-period ETF portfolio benchmark, dXPP's backward pass is reported 343x faster than dQP at horizon 200, and it is 4.2x to 9.2x faster on 1,000,000-variable projections. That makes tight-tolerance dXPP the strongest existing method. We beat it only if we reach the same final task metric, for example portfolio return or Sudoku error, in less total forward-plus-backward wall-clock time.
The most direct competitor is dXPP run at the same loose solver tolerance as our method, with identical $\delta$, solver, and hardware. This isolates the contribution of the correction steps. We beat it if, at matched tolerance, our gradients are at least an order of magnitude closer to the ground truth, and if this translates into better final task loss or faster convergence.
The third baseline is a PRDP-style schedule: loose-to-tight solver tolerance on dXPP, with the plateau-triggered controller tuned on the same budget of validation runs as ours. It is the alternative most likely to match us. We would beat it by reducing total solver time at equal final accuracy, especially late in training, when PRDP must solve tightly and we need not.
We also compare against FFOLayer, which avoids Hessians altogether, and against one-step differentiation through the solver's own last ADMM iteration, for OSQP run in a differentiable reimplementation. We include three ablations of our own. Corrections with $k=0$ recover loose dXPP. Newton corrections on the active-set KKT system without smoothing test whether smoothing is what handles degeneracy. A small learned residual network trained to map the residual $\nabla P_\delta(z_0)$ to a correction tests whether learning adds anything over analysis.
Why not just…?
Why not just tighten the solver tolerance? It works, and it is our ground truth. The question is cost. PRDP reports 62% to 81% savings from not solving precisely in iterative-physics training, and we expect a comparable regime for QP layers. Wall-clock curves against the tight-tolerance baseline settle this. If solver time is a small share of total training time on a benchmark, our method has nothing to offer there, and we will report that.
Why not just use a PRDP-style tolerance schedule? This is the real threat. Coarse gradients may be good enough early in training, and tight solves later may be cheap enough with warm starts. Our method differs because it keeps gradients accurate at loose tolerance throughout training, including the late phase. The matched-budget comparison above decides it. If the schedule alone matches us, the contribution shrinks to a controller that combines both, and we would say so.
Why not just run dXPP at loose tolerance, since smoothing already tolerates blurry active sets? Smoothing removes the classification step, but dXPP still evaluates its linear system at a point that is neither the true optimum nor the smoothed optimum. The implicit function theorem is therefore applied off its fixed point, and the error scales with solver tolerance. The $k=0$ ablation measures exactly this gap across tolerances on Maros-Meszaros.
Why not learn the correction, as residual-learning work does? A learned corrector needs training data, can violate constraints, and has no convergence guarantee. The Newton step on $P_\delta$ has all three properties for free. HyPINO's success means we should test the learned version, not dismiss it. The ablation compares correction quality at equal compute.
Proposed Experiments
The first decisive check is gradient accuracy. On all 129 Maros-Meszaros problems with randomly perturbed cost vectors, we plot relative gradient error against solver tolerance, sweeping tolerance from $10^{-2}$ to $10^{-9}$, $k$ from 0 to 3, and $\delta$ across several decades. The method succeeds if, for $k\ge 2$, error becomes flat in tolerance down to tolerance near $\delta$, consistent with the predicted basin condition. Loose dXPP and dQP should instead degrade or fail as tolerance loosens. We report degenerate problems separately, where we expect dQP's failures to cluster.
The second check is scaling. On the probability-simplex and bounded-difference projections up to 1,000,000 variables, we measure forward-plus-backward time at fixed gradient accuracy.
The third check is end-to-end learning. We train on multi-period ETF portfolio optimization with daily returns from 7 ETFs over 2011–2024, on Sudoku posed as a degenerate QP, and on the synthetic decision-focused QP up to dimension 1,000. We report final task metrics and wall-clock time to reach them. Each method gets three seeds and an equal hyperparameter search budget.
For the CSI 500 portfolio task, BPQP reports an annualized Sharpe ratio of 1.28, compared with 0.65 for two-stage methods. We will reproduce that number before claiming any comparison. If the reproduction fails, we report results only relative to our own baselines.
The headline would be equal task performance with markedly lower total solver time than tight-tolerance dXPP and the PRDP-style schedule. The decisive negative outcome would be loose dXPP matching our task metrics despite worse gradients. That would show downstream training does not need gradient accuracy in this regime.
References
- A Penalty Approach For Differentiation Through Black-box Quadratic Programming Solvers
- One-step differentiation of iterative algorithms
- PRDP: Progressively Refined Differentiable Physics
- Differentiation Through Black-Box Quadratic Programming Solvers
- A Fully First-Order Layer for Differentiable Optimization
- Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation
- Utility Decomposition with Deep Corrections for Scalable Planning under Uncertainty
- HyPINO: Multi-Physics Neural Operators via HyperPINNs and the Method of Manufactured Solutions
- BPQP: A Differentiable Convex Optimization Framework for Efficient End-to-End Learning