Solver-Aware Pruning of Latent Couplings in Irregular-Time Neural ODEs

Models that treat time as continuous are the natural fit for data that arrives at uneven intervals, such as ICU vital signs, glucose monitors, and wearable sensors. A neural ordinary differential equation (neural ODE) learns how a hidden state changes at every instant. A numerical solver then moves that state forward across whatever gap separates two observations. These models extrapolate well, but they are slow. The cost is set mostly by how many times the solver has to call the network, and pruning weights does little about it. Worse, pruning can make things slower: a sparsified vector field can develop sharp local features that force the solver to take more steps. We propose to prune something different. Instead of individual weights, we remove the couplings between groups of latent variables, meaning which parts of the hidden state are allowed to influence which others. We also let the solver itself decide how fast pruning may proceed.

The core claim is that for irregularly sampled data, the right target for sparsification is the interaction graph of the latent dynamics, and the right clock for the schedule is solver effort, not training step. We ramp up coupling sparsity gradually, as in standard progressive pruning. The ramp pauses, and pruned couplings are allowed to regrow, whenever the solver's evaluations per unit of integrated time start to climb. The result should be a model that is smaller and also measurably faster in wall-clock time. Its sparse latent graph should also split into weakly coupled blocks that evolve on different timescales. We test that last property directly, including on chemical systems whose true coupling graph is known.

What exists and what is missing

The base architecture is the ODE-RNN and Latent ODE of Latent Ordinary Differential Equations for Irregularly-Sampled Time Series. In these models the hidden state flows under a learned ODE between observations and is updated by a recurrent cell at each observation. We reuse this setup unchanged, apart from the vector field. That paper reports that the ODE-RNN takes about 60% more time than a GRU, and the Latent ODE roughly twice as long again. Multi-Time Attention Networks for Irregularly Sampled Time Series removes the solver altogether and reports more than 85x faster training per epoch. Its speed is the pressure that motivates this work.

Three lines of work already combine two of our ingredients, and each leaves a specific gap.

Sparse Flows prunes continuous-depth flows by repeated train–prune–retrain cycles on weight magnitudes, reaching 98% sparsity on static density benchmarks. It also observes that the solver can need more function evaluations above roughly 96% sparsity. That observation motivates our design: weight sparsity alone is not a speed-up for ODE models.

Automatic and Structure-Aware Sparsification of Hybrid Neural ODEs sparsifies the interaction graph of a hybrid physiological ODE with a fixed Group LASSO penalty, a regularizer that pushes whole groups of weights to zero together. It reaches RMSE 35.22 on T1DEXI. However, it starts from a hand-built mechanistic diagram of glucose physiology. We learn the graph from a generic latent space, and we replace the fixed penalty with a schedule.

Pruning via Sparsity-indexed ODE traces a continuous path from dense to sparse networks, which allows pruned masks to regrow. It was tested only on vision CNNs. We borrow its regrowth idea but drive the path with solver feedback.

On the scheduling side, To Prune, or Not to Prune supplies the cubic schedule used in gradual magnitude pruning. That schedule is the default we must beat.

Two papers constrain the design:

  • How to Train Your Neural ODE shows that penalizing the vector field's speed and the size of its Jacobian lowers solver evaluations with no pruning at all. Its method, RNODE, is therefore our most dangerous competitor.
  • Stiff Neural Ordinary Differential Equations documents gradient failure when timescales are widely separated. This warns us that any timescale separation our pruning produces must not make training stiffer.

Core mechanism

We split the latent state $h$ into $K$ blocks $h_1,\dots,h_K$. The vector field is written blockwise:

$$\frac{dh_i}{dt} = g_i\Big(h_i,\ \textstyle\sum_{j\neq i} m_{ij}\, W_{ij} h_j,\ t\Big).$$

Here $g_i$ is a small network that produces the rate of change of block $i$. $W_{ij}$ is the group of weights through which block $j$ enters it, and $m_{ij}\in[0,1]$ is a soft gate on that coupling. Setting $m_{ij}=0$ removes an entire off-diagonal block of the Jacobian. Removing whole blocks of the Jacobian, rather than scattered weights, is what gives real per-evaluation savings and a chance at timescale separation.

Each coupling gets an importance score from the group norm of $m_{ij}W_{ij}$, and the lowest-scoring couplings are gated off first.

The target coupling sparsity $s_t$ follows the cubic ramp of Zhu and Gupta, with one change: the ramp only advances while solver effort stays controlled. We measure

$$\rho_t = \frac{\text{solver function evaluations in the batch}}{\text{total integrated time in the batch}}.$$

The quantity $\rho_t$ is normalized by integrated time because irregular sampling makes gap lengths vary from batch to batch. Raw evaluation counts would conflate hard dynamics with long gaps.

If a moving average of $\rho_t$ exceeds $(1+\delta)$ times its value at the start of pruning, three things happen:

  1. The schedule pauses.
  2. The pruned couplings with the largest gradient scores are reopened, in the spirit of SpODE's regrowth.
  3. A Jacobian penalty of the RNODE form is applied only to the couplings that remain.

The only new hyperparameter is $\delta$, the tolerated increase in solver effort. We sweep it rather than tune it per dataset.

Research Program

The work proceeds in three stages.

The first stage establishes the tension empirically in the irregular-time setting. On Latent ODE and ODE-RNN models trained on MuJoCo hopper and PhysioNet 2012, we apply cubic unstructured magnitude pruning and record $\rho_t$ across sparsity levels. The goal is to see whether the rise in solver evaluations that Sparse Flows reported for static flows also appears here. If it does not, the solver-gating component loses its motivation, and we would say so.

The second stage builds the blockwise gated vector field and the solver-gated schedule. We implement them inside the Latent ODE codebase lineage with adaptive solvers, and we log wall-clock time, not only evaluation counts.

The third stage analyzes the learned graph. We ask whether the surviving blocks separate timescales, using the spread of eigenvalues of each block's local Jacobian. We then ask whether that separation can be cashed in. Blocks with no remaining cross-couplings in a given direction can be integrated with separate step sizes, and we measure how much further this cuts evaluations at test time.

Baselines

The naive reference point is a dense Latent ODE or ODE-RNN trained exactly as in Latent Ordinary Differential Equations for Irregularly-Sampled Time Series. It reports MSE 2.361 on PhysioNet interpolation versus 3.215 for the non-ODE baseline, and MSE 0.360 on MuJoCo with 10% of points observed. We rerun it in our codebase so that wall-clock numbers are comparable.

The strongest baseline for the question "can we make latent ODEs cheaper without losing accuracy" is RNODE-style kinetic and Jacobian regularization from How to Train Your Neural ODE, applied to the dense Latent ODE. That paper reports training time on CIFAR-10 flows falling from 91.3 to 31.8 hours. For irregular time series we found no reported result on Lacuna for RNODE, and no reported solver-evaluation counts for Latent ODE on PhysioNet. We will produce both. It is run with the same data splits and solver tolerances, and an equal hyperparameter search budget: the same number of trials over its penalty weights as we spend over $\delta$. Beating it means lower median $\rho_t$ and wall-clock per epoch at interpolation or extrapolation MSE within one standard error, over five seeds.

For raw speed on clinical classification, the strongest competitor is mTAN from Multi-Time Attention Networks for Irregularly Sampled Time Series, which reports 0.858 mortality AUC on PhysioNet 2012 and 0.910 accuracy on Human Activity. We do not expect to beat its speed. We report it as the bar our method must approach on accuracy, and we compare against it on extrapolation, where ODE models have their advantage.

On T1DEXI, the reference is HGS at RMSE 35.22, with the unreduced mechanistic model at 36.19 and an LSTM at 40.69 from Automatic and Structure-Aware Sparsification of Hybrid Neural ODEs. Coming within their range without a mechanistic prior would be the relevant success.

Three ablations isolate the key ingredients, each at matched final parameter count:

  • Coupling versus weight sparsity: cubic unstructured weight pruning of the same dense model.
  • Solver gating versus a plain schedule: our blockwise coupling pruning on a plain cubic ramp without solver gating.
  • Schedule versus fixed penalty: a static Group LASSO on the couplings, in the style of Zou and Tian.

The Sparse Flows repeated-pruning procedure is included once on MuJoCo, and we report its total training compute.

Why not just…?

Why not just regularize the dense vector field with RNODE and skip pruning? This is the real threat. The RNODE paper shows its Jacobian penalty directly lowers solver effort, and our method reuses that penalty. If dense RNODE matches our $\rho_t$ and accuracy, the only remaining benefit of pruning is parameter compression, and the solver-gating claim fails. The RNODE baseline and the "coupling pruning without gating" ablation together settle this.

We expect pruning to add something for one reason. RNODE shrinks the Jacobian's magnitude, but it leaves the Jacobian dense. Only structural zeros enable per-block step sizes and cheaper evaluations. The test-time experiment with separate step sizes per block is where that difference should become visible.

Why not just use mTAN, which is already far faster? For PhysioNet mortality classification, mTAN may simply remain the better tool, and we will not claim otherwise. The case for continuous dynamics rests on extrapolation and rollout stability. Evidence for that includes the Latent ODE's 0.360 versus 2.454 MSE on sparsely observed MuJoCo, and the gap between continuous models and a Transformer on T1DEXI (35.22 to 36.19 versus 46.29). We therefore report mTAN on MuJoCo extrapolation and T1DEXI as well. If mTAN matches us there too, the practical value of this work narrows to settings that need an explicit dynamical model.

Why not just apply the Zhu–Gupta cubic schedule to the ODE's weights? That is the obvious combination, and it is our first ablation. Sparse Flows' evidence that high weight sparsity can raise solver evaluations suggests it will compress parameters without cutting runtime. Stage one tests whether this happens under irregular sampling.

Why not just train a smaller dense latent ODE? Zhu and Gupta report that pruned large models beat small dense ones at equal parameter count, for example a 90% sparse LSTM that outperforms a medium dense model on PTB. Sparse Flows reports the same for continuous flows. Neither measured solver cost, though, and a narrow dense ODE might also need fewer evaluations. We include width-matched and parameter-matched dense models, and compare them on accuracy and on $\rho_t$.

Proposed Experiments

We evaluate on five settings:

  • MuJoCo hopper: 10% and 30% of points observed, with interpolation and extrapolation MSE.
  • PhysioNet 2012: interpolation MSE at 50–90% observed, and mortality AUC.
  • Human Activity: classification accuracy.
  • T1DEXI: RMSE and peak RMSE.
  • ROBER and POLLU: the stiff chemical systems from Stiff Neural Ordinary Differential Equations, resampled at irregular times.

For every model we report accuracy, $\rho_t$, wall-clock per epoch and at inference, parameter count, and retained coupling fraction. All results are means over five seeds.

Four checks decide the outcome.

The first check is whether, at matched accuracy, our method has lower $\rho_t$ and wall-clock than dense RNODE on MuJoCo and PhysioNet. If not, the solver-gated structural claim is false.

The second is whether unstructured cubic pruning shows rising $\rho_t$ at high sparsity while our gated schedule holds it within $(1+\delta)$.

The third concerns structure recovery. On POLLU, whose true reaction graph is sparse, we treat the latent ODE's state as the observed species. We then measure how well the surviving couplings match the true graph, using precision and recall, compared with static Group LASSO. This checks whether the schedule finds real structure, or merely any structure that happens to predict well. Zou and Tian flag this identifiability concern: many different graphs can give the same predictions.

The fourth is whether integrating decoupled blocks with separate step sizes lowers test-time evaluations further without raising MSE. That would show the timescale separation is real and usable, not just a descriptive statistic.