Differentiating Bang-Bang Controllers Through Their Switching Times

Many optimal controllers for time-critical or fuel-critical tasks are bang-bang: the actuator is always at full throttle in one direction or the other, and the whole control signal is described by a short list of instants at which it flips. Minimum-time drone flight, battery arbitrage and quantum gate pulses are all examples. These controllers are hard to put inside a learning pipeline. The control signal is piecewise constant, so its derivative with respect to any learnable parameter is zero almost everywhere and infinite at the jumps. Existing differentiable controllers avoid the problem by adding a quadratic control penalty or a log barrier. That penalty removes the bang-bang structure the task calls for, and as it is shrunk toward zero the backward computation becomes numerically unstable.

This note proposes to differentiate the switching times rather than the control signal. Once the sequence of flips is known, the optimal control problem becomes a small finite-dimensional problem over those few times. Its optimality condition has a clean form: the switching function, a scalar computed from the adjoint (costate) trajectory whose sign decides the throttle, must vanish at each switch. This small problem has genuine, invertible curvature even though the original problem has none in the control. The implicit function theorem can therefore be applied to it exactly. The result would be a differentiable layer that outputs true bang-bang or bang-off-bang controls in the forward pass and returns exact, well-conditioned gradients in the backward pass. This would allow system identification, cost learning and design tuning to be done directly on the controller that will actually be deployed.

Core Claim

The zero-curvature obstacle is a matter of coordinates. Current differentiable control methods all require the Hamiltonian (the scalar that optimal control minimizes pointwise) to be curved in the control, and they fail when it is linear. Differentiable MPC (Amos et al.) differentiates the KKT conditions of an iLQR subproblem. Pontryagin Differentiable Programming, or PDP (Jin et al.), solves an auxiliary linear-quadratic system built from the Hamiltonian's second derivatives. DiffMPC (Adabag et al.) runs a conjugate-gradient solver that needs a positive definite control weight. For a Hamiltonian linear in $u$, the second derivative $\nabla^2_{uu}H$ is identically zero and all three methods break. The curvature has not disappeared, however. It has moved into the switching times.

Fix a sign pattern $s$ and $M$ switching times $\tau=(t_1,\dots,t_M)$. Write $J(\tau,\theta)$ for the cost of the resulting trajectory under parameters $\theta$, with terminal constraints $\Phi(x(T),\theta)=0$ and multipliers $\nu$. A standard calculation gives $\partial J/\partial t_i = (u_i^- - u_i^+)\,\sigma(t_i)$, where $u_i^\mp$ are the control values just before and after switch $i$ and $\sigma$ is the switching function. So stationarity in $\tau$ is the same statement as $\sigma(t_i)=0$ from Pontryagin's principle. Differentiating the stationarity and feasibility conditions gives the bordered system

$$ \begin{bmatrix} \nabla^2_{\tau\tau}\mathcal{L} & \nabla_\tau \Phi^\top \\ \nabla_\tau \Phi & 0 \end{bmatrix} \begin{bmatrix} \partial \tau^*/\partial\theta \\ \partial \nu^*/\partial\theta \end{bmatrix} = -\begin{bmatrix} \nabla^2_{\tau\theta}\mathcal{L} \\ \nabla_\theta \Phi \end{bmatrix}, $$

where $\mathcal{L}=J+\nu^\top\Phi$ is the Lagrangian. In parametric optimal control it is classical that positive definiteness of $\nabla^2_{\tau\tau}\mathcal{L}$ on the constraint tangent space, together with strict switching ($\dot\sigma(t_i)\neq 0$), is the natural second-order sufficient condition for a bang-bang extremal. Under those conditions the system above is invertible. The system has size $M$ plus the number of terminal constraints. For a reverse-mode gradient, one adjoint ODE sweep supplies the needed matrix-vector products.

The contribution is not the single-switch sensitivity formula, which the file already sketches. It is a working differentiable layer built on this reduced problem. The layer accepts any forward solver, handles terminal constraints and free final time, extends to maximum hands-off (bang-off-bang) problems, and behaves predictably when the switching structure changes during learning. The layer will be tested head-to-head against smoothing, which is the approach it has to beat.

Research Program

Forward pass. The forward pass can use any solver that produces an approximate bang-bang solution: a closed form, a bisection search like that of Westenberger et al., or a coarse-grid linear program. We read off the sign pattern from its output. Then we refine the switching times by a few Newton steps on the reduced stationarity conditions. The refinement step matters because implicit gradients are only exact at a converged solution.

Maximum hands-off extension. The same construction covers bang-off-bang control, where an $L^1$ fuel cost adds a zero-throttle "off" state. Here switches occur where $|\sigma| = 1$ rather than where $\sigma=0$. Chatterjee et al. showed that the exact problem has bang-off-bang structure for linear time-invariant systems, and that the $L^1$ relaxation fails only in singular cases.

Backward pass. The backward pass solves the bordered system above as a vector-Jacobian product. The terms $\nabla_\tau J$ and $\nabla_{\tau\theta}J$ come from the state and adjoint trajectories between switches. Its cost scales with $M$, not with the time grid.

Structure changes. The switching structure changes when two switches merge, a new switch appears, or $\dot\sigma(t_i)\to 0$. At such points the gradient is only one-sided. The layer will detect near-degeneracy from $|\dot\sigma(t_i)|$ and from gaps $t_{i+1}-t_i$ falling below a tolerance. It will then return the one-sided gradient of the current structure and flag the event. We will measure how often these events occur during training. We will also test a fallback that applies small smoothing only in a local window around a degenerate switch.

Scope. Singular arcs, where $\sigma$ vanishes on an interval, are explicitly out of scope. They are detected and reported, not handled.

Baselines

Naive reference. A fixed-grid linear program solved with a differentiable convex layer (Agrawal et al.), which differentiates the KKT residual map of cone programs. On a grid, the parameter gradient flows only through the one cell that straddles each switch. It should therefore be both starved and accurate only to about the grid step. This baseline shows what "just use existing layers" delivers. Beating it means lower gradient error than the grid provides at equal wall-clock time.

Strongest competitor: smoothing. Smoothing covers three variants: a quadratic regularizer $\epsilon\|u\|^2$ fed to DiffMPC or PDP, log barriers as in Safe PDP, and the decaying-barrier homotopy of Kordabad et al.. The homotopy variant was designed specifically for the gradient-starvation problem addressed here. For fairness, each smoothing method gets the same model, data and wall-clock budget. Its smoothing parameter or decay schedule will be tuned by grid search with the same number of trials we spend on our own tolerances. Every learned parameter will be evaluated by deploying the exact bang-bang controller, not the smoothed one. Beating smoothing means one of two outcomes. The first is equal-or-better deployed performance with fewer iterations. The second is equal iterations with no train-deploy gap. The number to match on battery arbitrage is Kordabad et al.'s report that homotopy-smoothed MPC policy gradient converged in about 200 RL steps, with parameter trajectories matching the dynamic programming ground truth.

Unrolled switching-time optimization. This baseline uses automatic differentiation through the iterations of the SLSQP switching-time solver of Fei et al.. It shares our reduced coordinates, so any gain must come from implicit versus unrolled differentiation. It gets the same forward solution, and its number of unrolled iterations is chosen to match our forward cost.

Model-free comparison. On cartpole, a Bernoulli policy trained with a straight-through estimator (Seyde et al.) serves as the model-free comparison.

Ablations. The first ablation removes Newton refinement, leaving implicit differentiation at a loosely converged solution. The second disables the degeneracy handling. Together they isolate which ingredient carries the gains.

Why not just…?

Why not just add a tiny quadratic penalty and use DiffMPC or PDP? This is the serious threat. The smoothed problem's gradient may converge to ours as $\epsilon\to 0$. Our argument rests on two points. First, the backward system's conditioning worsens as the smoothing shrinks, which Safe PDP itself reports for its barrier parameter. Second, any finite $\epsilon$ yields a controller that is not bang-bang, and in the hands-off case it loses the "off" intervals: Chatterjee et al.'s double integrator example stays at zero for roughly 40% of the horizon under the exact formulation, while smooth surrogates are nonzero throughout. The first experiment below measures directly whether a best-tuned $\epsilon$ reaches our gradient accuracy at acceptable conditioning. If it does, our layer's advantage shrinks to convenience, and we will report that.

Why not just unroll the switching-time optimizer? In switching-time coordinates the problem has only a handful of variables, so the usual memory argument against unrolling is weak. Scaling results such as PDP's 1000x speedup over unrolled differentiable MPC apply to grid-sized problems, not this one. The honest case for implicit differentiation here is different. It is exact at convergence regardless of how many iterations the solver used. It also works with forward solvers that have no differentiable iterations at all, such as closed forms or bisection. The unrolled-SLSQP baseline tests whether this matters in practice.

Why not just use Bernoulli policies with a straight-through estimator? They work surprisingly well on DeepMind Control Suite, but they chatter on cartpole balance, and their gradient ignores the dynamics. Model-based differentiable control needs far less data: Amos et al. converged on pendulum imitation with about 10 trajectories where a multilayer network needed over 100. We include this comparison mainly on sample efficiency and steady-state chattering. We do not expect to beat it on reward for every task.

Why not just take finite differences, since parameters are few? For system identification with two or three parameters, finite differences are a legitimate competitor. We use them as the ground-truth gradient. We will report the parameter count at which they stop being competitive in cost.

Proposed Experiments

Gradient accuracy. On double integrator and linear time-invariant time-optimal and hands-off problems, analytic sensitivities are available. We compare every method's gradient against high-precision finite differences of the exact problem, reporting cosine similarity and relative error as functions of the smoothing parameter and grid step. We also record condition numbers of each backward system. The decisive check is whether our layer's error is flat and near machine tolerance while smoothing trades error against conditioning. If best-tuned smoothing matches us on both, the central claim fails.

Battery arbitrage. We learn the economic parameters of the bang-off-bang MPC policy in Kordabad et al.'s state-of-charge model, with the same LSTD outer loop and the same noise. We compare RL steps to convergence against the roughly 200 they report, and measure distance to the dynamic programming parameters.

Quadcopter system identification and tuning. We use a simulated version of Westenberger et al.'s drag-aware quadcopter model. Their bang-bang MPC flew 17.5% faster than a tuned PID controller at 512 Hz. We identify the drag and transition-delay parameters end to end from bang-bang flight logs, and evaluate lap time and waypoint error when the exact controller runs on held-out parameters. No reported learned-parameter result was found on Lacuna for this task, so the comparison is internal.

Quantum stretch goal. We differentiate optimal switching times with respect to Hamiltonian coupling parameters on Fei et al.'s spin-system energy tasks. This enables robust pulse design against parameter uncertainty. No reported result on Lacuna covers gradients with respect to exogenous parameters, so we compare against their fixed-parameter fidelity.

Structure-change events. In every experiment we log degeneracy events and one-sided gradient uses. This tests whether structure changes are rare enough in practice for the layer to be dependable.