Decoupling Forward and Backward Passes in Differentiable Optimization

This direction is split into other directions: solver-agnostic differentiation. Its last version stays readable and citable here.

Version 1 of 127 papers44 conceptsfrozen 2026-10-04

In traditional deep learning, a "layer" is an explicit mathematical formula—like a matrix multiplication followed by a ReLU. However, many real-world problems are better described as optimization problems: finding the best configuration of variables given a set of constraints. Integrating these problems as "optimization layers" allows neural networks to reason about physics, logic, and geometry.

The challenge is that standard backpropagation requires every step of a calculation to be recorded in a computational graph. For complex optimization solvers, this "unrolling" is memory-intensive, slow, and often impossible if the solver is a proprietary "black box" written in C++ or Fortran.

A growing body of research has converged on a powerful solution: decoupling. By separating the forward pass (finding the solution) from the backward pass (calculating the gradient), researchers can use the world’s fastest optimization solvers while maintaining exact, efficient gradients for training.

The Problem with Unrolling

To differentiate through an optimization problem, the naive approach is to "unroll" the solver's iterations. If a solver takes 100 steps to find a minimum, backpropagation must traverse those 100 steps in reverse.

As James Kotary and Ferdinando Fioretto point out in Analyzing and Enhancing the Backward-Pass Convergence of Unrolled Optimization, unrolling is essentially an inefficient way to solve a linear system. It couples the training process to the specific internal mechanics of the solver. If you want to use a high-performance, non-differentiable solver like Gurobi or MOSEK, unrolling simply isn't an option.

The Intuition of Implicit Differentiation

The core insight of the decoupling movement is that we don't need to know how a solver found the answer; we only need to know where it landed.

If an output $y$ is the solution to an optimization problem parameterized by $x$, then at the optimal point, certain conditions must be true (such as the gradient being zero or the KKT conditions being satisfied). Qianggong Zhang and colleagues describe this in Implicitly Defined Layers in Neural Networks: as long as we can define the "equilibrium" or optimality condition, we can use the Implicit Function Theorem to calculate how the solution $y$ changes as $x$ changes.

This allows for a modular architecture:

  1. Forward Pass: Use any specialized, black-box solver to find the solution.
  2. Backward Pass: Solve a separate, analytical linear system derived from the optimality conditions.

Modular and Black-Box Solvers

This decoupling enables the use of industrial-grade tools. Connor W. Magoon and Shahar Z. Kovalsky introduced dQP in Differentiation Through Black-Box Quadratic Programming Solvers, a framework that treats Quadratic Programming (QP) solvers as black boxes. By looking at the "active set" of constraints at the solution, they can compute gradients without ever seeing the solver's internal code.

Similarly, Mathieu Blondel and his team developed a framework for Efficient and Modular Implicit Differentiation that allows researchers to define optimality conditions in Python while using high-performance C++ solvers for the heavy lifting. This modularity means that if a faster solver is invented tomorrow, it can be dropped into the neural network without changing the differentiation logic.

Folded Optimization: The Best of Both Worlds

In Folded Optimization for End-to-End Model-Based Learning, James Kotary and My H. Dinh formalize this decoupling. They prove that backpropagating through an unrolled solver is mathematically equivalent to solving a specific linear system at its fixed point.

By "folding" the optimization, they allow the forward pass to be handled by arbitrary solvers, while the backward pass uses efficient numerical methods (like Krylov subspace methods) to solve the resulting linear system. This approach is not only faster but also more stable, avoiding the "gradient explosion" or vanishing problems common in unrolled iterations.

Stability and Speed in Practice

The benefits of decoupling extend beyond just speed. In Backpropagation-Friendly Eigendecomposition, Wei Wang and Pascal Fua demonstrate a hybrid approach for eigendecomposition. They use a precise algorithm for the forward pass but a different, iterative proxy for the backward pass to avoid numerical instabilities when eigenvalues are nearly identical.

Recent advancements like BPQP by Jianming Pan and dXPP by Yuxuan Linghu continue to push this boundary, achieving up to 300x speedups by using smoothed penalty approaches and first-order information to simplify the backward pass.

By decoupling the "how" of the solution from the "what" of the gradient, these researchers have made it possible to embed complex, large-scale mathematical reasoning directly into the heart of deep learning architectures.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decoupling_forward_and_backward_passes_i_e9462574,
  title = {Decoupling Forward and Backward Passes in Differentiable Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupling-forward-and-backward-passes-in-differentiable-optimization/rdr_e9462574ff044a8cb88c1ecc2c8399cc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_decoupling_forward_and_backward_passes_i_e9462574_v1,
  title = {Decoupling Forward and Backward Passes in Differentiable Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupling-forward-and-backward-passes-in-differentiable-optimization/rdr_e9462574ff044a8cb88c1ecc2c8399cc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}