When optimizing a system governed by differential equations—whether training a continuous-depth neural network or tuning a climate simulation—we must compute how a final objective changes with respect to the system's initial parameters. This sensitivity analysis relies on the adjoint method, a mathematical technique that propagates gradients backward through time.
However, a fundamental fork in the road exists for researchers implementing this method: Do you derive the mathematics of the gradient in the continuous world first and then solve it numerically, or do you differentiate the actual steps taken by your numerical solver?
The choice between these two paradigms—optimize-then-discretize (the continuous adjoint approach) and discretize-then-optimize (the discrete adjoint approach)—is the difference between optimizing an approximation of your system and obtaining mathematically exact gradients of your actual computation.
The Mismatch in Continuous Adjoints
The traditional continuous adjoint method operates under the optimize-then-discretize philosophy. You begin with a continuous forward differential equation, derive a continuous backward differential equation for the gradients, and hand both to a numerical solver. As noted by Gao et al. in their study of Neural ODEs (paper), it is often theoretically unclear whether these continuous approximations reliably guide an optimization algorithm toward the true global optimum of the system.
The core issue is a subtle but destructive numerical mismatch. A numerical solver does not trace a perfectly continuous path; instead, it takes discrete steps, introducing tiny truncation errors at every interval. When a continuous adjoint equation is solved backward in time, it assumes a smooth trajectory. Because the backward pass does not "know" about the specific numerical errors made during the forward pass, the forward and backward paths diverge.
As Matsubara et al. explain (paper), this discrepancy introduces numerical drift that can cause the gradient calculation to diverge entirely from the true gradient of the discrete computation. In complex fluid dynamics or wave simulations, this mismatch can manifest as severe numerical instabilities or completely incorrect descent directions, causing optimization algorithms to fail, a phenomenon documented by Funke et al. (paper). Furthermore, Albi et al. demonstrate that high-order numerical schemes can suffer from "order reduction" when continuous adjoints are applied, stripping the solver of its precision (paper).
Optimize-then-Discretize (Continuous Adjoint):
Continuous Physics ───(Derive Adjoint)───> Continuous Adjoint
│ │
(Discretize) (Discretize)
▼ ▼
Discrete Forward ────────────────────────> Mismatched Gradients
Discretize-then-Optimize (Discrete Adjoint):
Continuous Physics
│
(Discretize)
▼
Discrete Forward ───(Differentiate Code)───> Exact Gradients
The Discretize-then-Optimize Paradigm
The discrete adjoint approach flips the sequence: you discretize the forward system first, and then differentiate the resulting numerical steps. By treating the computer code of the forward solver as the ground truth, the discrete adjoint method accounts for every single approximation, rounding error, and step-size adjustment made during the simulation.
According to Ghelichkhan et al., deriving the adjoint directly from the discretized equations ensures that the calculated gradients are mathematically exact to floating-point precision for that specific numerical model (paper). This exactness guarantees a mathematically consistent descent direction. As Einkemmer et al. emphasize, because the adjoint is tailored to the exact implementation of the forward model, it eliminates the convergence issues that plague continuous approximations (paper).
This precision is particularly vital when the optimization objective interacts with the internal logic of the solver itself. For instance, adaptive step-size solvers use internal heuristics to determine when to grow or shrink a time step. Pal et al. show that discrete sensitivity analysis provides superior numerical stability because backpropagating through the actual solver steps is the only way to accurately capture the gradients of these internal solver heuristics (paper).
Exact Gradients in Practice
To see why this works, consider how a discrete solver updates a state $u_n$ to $u_{n+1}$ via a choice of function $f$:
$$u_{n+1} = u_n + \Delta t \cdot f(u_n, \theta)$$
In the discrete adjoint framework, we view this as a chain of composite functions, much like layers in a deep neural network. When we compute the gradient of a loss $L$ with respect to parameters $\theta$, we apply the chain rule directly to this sequence of operations:
$$\frac{\partial L}{\partial \theta} = \sum_{n} \frac{\partial L}{\partial u_{n+1}} \frac{\partial u_{n+1}}{\partial \theta}$$
Because we are differentiating the exact algebraic operations performed by the machine, there is no room for a physics-vs-simulation mismatch. Melchers et al. highlight that this yields higher accuracy than continuous methods because it provides the exact gradients of the numerical loss function, bypassing the approximation errors introduced when the forward and backward passes utilize different discretization schemes (paper).
This strategy is widely applicable across domains:
- Generative Modeling: Köhler et al. leverage this by unrolling fixed-grid ODE solvers into a sequence of layers, allowing direct backpropagation to improve training stability and convergence speed in normalizing flows (paper).
- Particle Simulations: For high-dimensional systems like the nonlinear Boltzmann equation, Caflisch et al. demonstrate that deriving adjoint sensitivities directly from the discrete steps of a simulation avoids the intractable task of solving high-dimensional continuous adjoint PDEs (paper).
- Physical Control: In complex engineering tasks, such as modeling porous media flow or beam dynamics, Kumar et al. (paper) and Sato Martín de Almagro et al. (paper) show that discretizing the adjoint variables using the exact same numerical framework as the state variables preserves mathematical consistency and guarantees algorithm convergence under physical constraints.
Bridging the Gap: Reversibility and Structure Preservation
If the discrete adjoint method is so robust, why isn't it used exclusively? The primary drawback is memory. While the continuous adjoint method can solve equations backward in time without storing intermediate states, standard discrete backpropagation requires caching the entire forward trajectory in memory.
To overcome this constraint, recent research focuses on aligning the two approaches by designing solvers where the discrete backward pass is the exact algebraic inverse of the forward pass.
Gholami et al. developed ANODE, a framework that pairs a discretize-then-optimize approach with a checkpointing strategy (paper). By saving specific snapshots of the forward pass and recomputing short trajectories in reverse, they achieve unconditionally accurate discrete gradients with only a fraction of the memory cost.
In the stochastic domain, Kidger et al. introduced algebraically reversible solvers for Neural Stochastic Differential Equations (SDEs) (paper). By ensuring that the numerical backward pass perfectly retraces the forward steps, they eliminate the numerical drift that causes gradient instability in continuous-time stochastic models.
From a geometric perspective, Tran et al. demonstrate that leveraging structure-preserving integrators (such as those built on Lie groups) can force discretization and adjoint differentiation to commute perfectly (paper). When this commutativity is achieved, the discrete gradient becomes identical to the discretized version of the continuous gradient, offering a unified path forward for high-accuracy optimization in robotics and deep learning.
Key Foundational Work
Reconstructing wave profiles from inundation data (2017) — contributed 2 concept elements to this direction.
Mixed and discontinuous finite volume element schemes for the optimal control of immiscible flow in porous media (2018) — contributed 2 concept elements to this direction.
Linear multistep methods for optimal control problems and applications to hyperbolic relaxation systems (2018) — contributed 1 concept elements to this direction.