Scaling Differentiable Optimization in Neural Networks

351 concepts285 papers4 proposals

Deep learning excels at pattern recognition, but real-world tasks often require making constrained decisions based on those patterns. This requirement has driven the rise of Decision-Focused Learning (DFL) and end-to-end constrained optimization architectures, as surveyed by Ferdinando Fioretto et al. in End-to-End Constrained Optimization Learning: A Survey. By embedding mathematical optimization solvers directly into a neural network's training loop, models can learn to minimize real-world "regret" rather than simple statistical error.

However, this integration introduces a massive computational bottleneck. As Jayanta Mandi et al. point out in Smart Predict-and-Optimize for Hard Combinatorial Optimization Problems, invoking a full optimization solver for every training instance in every epoch makes scaling to large datasets incredibly difficult. To make these architectures practical, researchers must understand why this overhead occurs and deploy clever strategies to bypass it.

The Root of the Bottleneck

The computational overhead of solver-in-the-loop architectures manifests during both the forward and backward passes of training.

In the forward pass, the network must wait for a complete optimization problem to resolve for every single data point or batch. When dealing with discrete or combinatorial problems—such as the mixed-integer linear programs used by Elias Benussi et al. in Individual Fairness Guarantees for Neural Networks—the solver must tackle NP-hard problems at every iteration. As Marianne Defresne et al. observe in Learning to Reason: Predict-then-Optimize vs Predict-and-Optimize, this requirement makes end-to-end training prohibitively slow as the number of decision variables increases.

The backward pass introduces an even steeper hurdle: computing gradients through the solver. Standard automatic differentiation typically relies on unrolling the internal iterative steps of the solver. As Brandon Amos et al. explain in OptNet: Differentiable Optimization as a Layer in Neural Networks, unrolling tracks every intermediate operation, which consumes massive amounts of memory and introduces severe computational overhead.

Alternatively, researchers can use implicit differentiation to compute gradients analytically through the solver's optimality conditions (such as the KKT conditions). While this avoids tracking intermediate steps, it requires constructing and inverting large matrices. Kai Wang et al. demonstrate in Automatically Learning Compact Quality-aware Surrogates for Optimization Problems that this process scales cubically ($O(N^3)$) with the number of decision variables, creating a severe scalability bottleneck. Léonard Tschora notes in Electricity Price Forecasting based on Order Books: a differentiable optimization approach that these delays prevent optimization-aware training from operating in complex, high-frequency environments.

To alleviate this overhead, the field has converged on four primary mitigation strategies.

Strategy 1: Differentiable Neural Surrogates

Instead of calling an exact mathematical solver during training, one intuitive solution is to replace the solver with a smooth, fast neural network proxy that mimics the optimization landscape.

[Input Data] ---> [Neural Network Predictor] ---> [Neural Surrogate] ---> [Dense Gradients]
                                                         ^
                                            (Trained to mimic true solver)

For example, Arman Zharmagambetov et al. introduce the LANCER framework in Landscape Surrogate: Learning Decision Losses for Mathematical Optimization Under Partial Information. LANCER trains a neural network surrogate to approximate the decision loss landscape, providing the dense gradients needed for training while drastically reducing the number of times the heavy-duty solver must be called.

Similarly, Lingkai Kong et al. propose SO-EBM in End-to-end Stochastic Optimization with Energy-based Model, which uses Energy-Based Models as a differentiable surrogate to eliminate exact solver calls entirely during gradient steps. Rather than approximating the loss landscape, Rares C Cristian et al. introduce ProjectNet in End-to-End Learning for Optimization via Constraint-Enforcing Approximators, replacing exact solvers with a differentiable, constraint-enforcing approximator based on Dykstra’s projection algorithm to accelerate training.

Strategy 2: Solution Caching and Contrastive Learning

Optimization problems across training epochs often hit identical or highly similar optimal solutions. Instead of re-solving the problem from scratch every time a parameter updates, models can maintain a cache of previously discovered solutions.

Jayanta Mandi et al. leverage this insight in Contrastive Losses and Solution Caching for Predict-and-Optimize. By treating cached solutions as an "inner approximation" of the search space, the authors use contrastive loss functions to train the model to rank the correct cached solution above suboptimal ones. As further shown by Maxime Mulamba et al. in Contrastive Losses and Solution Caching for Predict-and-Optimize, this combination bypasses repeated solver calls entirely during backpropagation, slashing training times by orders of magnitude while maintaining high decision quality.

Strategy 3: Reformulating the Loss Landscape

Rather than forcing the network to interact with a solver layer, researchers can reframe the learning objective so that the network naturally learns the structure of the downstream optimization task.

In Decision-Focused Learning: Through the Lens of Learning to Rank, Jayanta Mandi et al. reframe DFL as a Learning to Rank (LTR) problem. By training the model to correctly rank feasible solutions rather than computing exact solver gradients, they bypass the solver loop entirely during backpropagation.

Alternatively, Sanket Shah et al. propose Locally Optimized Decision Losses (LODLs) in Decision-Focused Learning without Decision-Making: Learning Locally Optimized Decision Losses. Instead of handcrafting differentiable surrogates, LODLs automatically learn convex, task-specific loss functions using a black-box optimization oracle. This decouples the forward training loop from active decision-making, enabling faster, highly parallelizable model training.

Strategy 4: Matrix-Free Adjoint Methods

For continuous optimization layers where iterative solvers must be used, the unrolling bottleneck can be mitigated by changing how gradients are derived mathematically.

Hrittik Roy et al. demonstrate this in Matrix-Free Least Squares Solvers: Values, Gradients, and What to Do With Them. By deriving custom gradients for matrix-free iterative solvers (such as LSMR) using the adjoint method, the authors eliminate the need to unroll the solver's internal steps. This technique bypasses excessive memory consumption and allows differentiable optimization layers to scale to architectures with millions of parameters.

The Path Forward

As Haoyu Geng notes in a comprehensive evaluation in Benchmarking PtO and PnO Methods in the Predictive Combinatorial Optimization Regime, while end-to-end decision-focused learning generally outperforms traditional two-stage pipelines, its computational overhead remains a critical barrier. The future of the field relies on decoupling heavy mathematical machinery from the gradient loop—whether through surrogates, caching, or matrix-free formulations—to make decision-aware machine learning scalable to real-world industrial problems.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 286 papers
  2. Version 1 · 2026-10-04 · 300 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_differentiable_optimization_overhead_e1b18050,
  title = {differentiable optimization overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/differentiable-optimization-overhead/rdr_e1b18050c767442981312b5d5aed813e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 2, frozen text):

@misc{lacuna_differentiable_optimization_overhead_e1b18050_v2,
  title = {differentiable optimization overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/differentiable-optimization-overhead/rdr_e1b18050c767442981312b5d5aed813e/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake