Kronecker-Sum Curvature Sharing for Federated Neural CDEs

Hospitals, wearables and sensor networks all collect time series that arrive at irregular moments, and they usually cannot pool that data in one place. Neural controlled differential equations are the natural model for such data. A neural CDE is a continuous-time analogue of a recurrent network: a small neural network defines how a hidden state changes in response to the incoming signal, and an ODE solver integrates it. These models are known to be slow and badly conditioned to train even on one machine. Curvature-aware optimizers, which rescale each update using an estimate of how the loss surface bends, fix much of this centrally. Federated training, where many sites update a shared model without sharing raw data, adds client drift and a tight communication budget on top of that slow optimization. This note asks whether the curvature information can be shared across sites cheaply and correctly. The target is federated models that reach centralized accuracy in far fewer communication rounds.

The central point is specific to continuous-time models trained on heterogeneous sites. Sites differ mainly in how they observe time: sampling rates, missingness, observation horizons. In a neural CDE, the input that drives the hidden state is built from the observed path, so these temporal differences change the structure of each site's curvature, not just its scale. The obvious recipe averages one pair of per-layer curvature factors across sites. That recipe silently assumes the sites look alike. We propose instead that the server maintain a sum of a few Kronecker-factored curvature terms, one per group of temporally similar sites. The terms are learned over rounds from compressed updates, and the preconditioner can still be inverted exactly for two terms.

Core Claim

We make one claim. On neural CDE training, a server-side preconditioner built as a small sum of Kronecker products, grouped by each site's sampling regime, will reach a target accuracy in fewer communication rounds and bits than two alternatives. The first is federated first-order training. The second is the naive federated extension of centralized second-order Neural ODE training, which uses a single averaged Kronecker factor pair. The advantage should grow with temporal heterogeneity across sites and vanish when sites are identically distributed. That last prediction is what makes the claim falsifiable.

For one layer of the vector field, site $i$ produces an input-side factor $A_i$ and an output-side factor $G_i$. Here $A_i$ is the covariance of the layer's inputs accumulated along the integrated trajectory, and $G_i$ is the covariance of the backpropagated sensitivities. The curvature the server actually wants is the data-weighted sum of per-site Kronecker products. We approximate that sum with $R$ group terms:

$$F \;\approx\; \sum_{r=1}^{R} \bar A_r \otimes \bar G_r + \lambda I,$$

where $\bar A_r, \bar G_r$ are sample-weighted averages over the sites in group $r$ and $\lambda$ is damping. The naive recipe is the case $R=1$, and it misses the cross-terms $\sum_i A_i \otimes G_i - (\sum_i A_i)\otimes(\sum_i G_i)/n$. These cross-terms are zero only when sites share curvature structure. For $R=2$ the inverse is exact and cheap: a generalized eigendecomposition of $(\bar A_2, \bar A_1)$ and $(\bar G_2, \bar G_1)$ diagonalizes both terms simultaneously. For larger $R$, the server runs a few conjugate-gradient steps using Kronecker matrix-vector products.

Where This Sits in Prior Work

The method's backbone comes from Second-Order Neural ODE Optimizer (SNOpt). SNOpt derives coupled backward ODEs that return gradients and Kronecker-factored curvature in a single adjoint pass at constant memory. It reports 95.77% versus SGD's 93.34% on SVHN, and a 20% runtime cut on CIFAR-10 from learning the integration horizon. SNOpt is strictly centralized. We reuse its curvature extraction through the ghliu/snopt patch to torchdiffeq, and change what happens to the factors after they leave each site.

Distributed Second-Order Optimization using Kronecker-Factored Approximations shows that Kronecker-factored curvature tolerates staleness and scales across datacenter workers, trained GoogLeNet and ResNet-50 on ImageNet twice as fast as tuned SGD. Its workers see identically distributed data, so a single averaged factor pair is appropriate there. We supply a reason that assumption breaks under temporal non-IID data.

Two further papers supply the communication machinery. FedNL has clients send compressed differences between their current curvature and a running estimate, so the server gradually learns the curvature. It reports a $10^{-15}$ optimality gap where first-order baselines stall near $10^{-3}$ to $10^{-5}$. FedNS sketches square-root Hessians and reaches high precision on SUSY within 20–50 rounds while FedAvg and FedProx do not converge in 100. Both work only on convex logistic regression with full Hessians of static models. We borrow FedNL's "learn the curvature via compressed shifts" idea and apply it to the small Kronecker factors, not to a $d \times d$ matrix.

Optimal Shrinkage for Distributed Second-Order Optimization constrains the design. It shows that averaging local inverse curvatures is biased and proposes a scalar correction. We therefore aggregate factors and invert only at the server, which avoids that bias by construction. A GIANT-style local-inverse variant, with and without the shrinkage fix, serves as a comparison arm.

The continuous-time analyses of federated optimization, Continuous-Time Analysis of Federated Averaging and FedECADO, use ODEs to describe optimizers. They do not train ODE models and are orthogonal to this work.

Research Program

The first component is the local computation. Each site trains a neural CDE in the style of Neural Controlled Differential Equations for Irregular Time Series. Once the control path is interpolated, a neural CDE is an ordinary ODE with vector field $f_\theta(z)\,dX/dt$, so SNOpt's adjoint curvature pass applies with modest adaptation. Every $k$ rounds, each site sends its gradient plus compressed shifts of its per-layer factors, using Top-$K$ or low-rank compression as in FedNL. These factors are small for the vector-field networks used here: a layer of width 128 has 128×128 factors. Communication is therefore dominated by the gradient-sized vectors, not by curvature.

The second component is grouping. Before training, each site reports a cheap descriptor of its sampling regime: median inter-observation interval, missingness rate and horizon length. The server clusters sites into $R$ groups on these descriptors. A learned alternative clusters sites on the distance between their factors after the first curvature round. Groups are fixed after a warm-up period so that the running factor estimates stay stable.

The third component is the server step. The server forms the Kronecker-sum preconditioner, inverts it exactly for $R=2$ or with conjugate gradients for $R>2$, and applies a damped, line-searched step to the aggregated gradient. The line search mirrors FedNL-LS, because the neural CDE loss is non-convex and far from any regime covered by the convex theory.

The fourth component is diagnostics. For small models we can afford the exact Fisher of the vector field at the server in simulation. That lets us measure directly how much curvature the $R=1$ approximation loses as heterogeneity grows, and whether the Kronecker-sum closes that gap. This measurement is separate from end-to-end accuracy and tests the mechanism itself.

Baselines

The naive reference point is FedAvg with local Adam on the same neural CDE, using the adjoint method in torchdiffeq. This is what practitioners would do today. We also run FedProx at the same settings. Both get the same local epochs, batch size, rounds and a matched hyperparameter search budget (the same number of trials over learning rate, local steps and proximal weight). No reported result was found on Lacuna for federated training of neural CDEs or Neural ODEs on any benchmark in the file. We therefore anchor to centralized numbers: the neural CDE paper reports 98.6% accuracy on CharacterTrajectories with 70% missing data, AUC 0.880 on PhysioNet sepsis, and 89.8% on Speech Commands. "Beating" a federated baseline means reaching 97% of the centralized accuracy we reproduce in fewer rounds and fewer uplink bits, with final accuracy no worse.

The strongest competitor is naive federated SNOpt. Each site computes SNOpt factors, the server averages them into one Kronecker pair, as in distributed K-FAC, and it takes the same damped, line-searched step. It receives identical compression, curvature refresh frequency and tuning budget, so the only difference from our method is $R=1$ versus $R>1$ with regime-based grouping. This is the decisive comparison. Our method beats it if it cuts rounds-to-target by a margin larger than the seed-to-seed spread under heterogeneous splits, while matching it under IID splits.

Two ablations isolate the key ingredient. The first uses $R>1$ with random site grouping. It separates "more curvature terms" from "terms aligned with temporal heterogeneity". The second is a GIANT-style arm in which sites invert their own factors locally and the server averages the resulting steps, with and without the Zhang–Pilanci shrinkage. It tests whether server-side inversion matters.

Local-only SNOpt with plain weight averaging, which shares no curvature at all, is treated as its own baseline below.

Why not just…?

Why not just run FedAvg with Adam on the neural CDE? It is simpler, and adaptive local optimizers already capture some diagonal curvature. The evidence against it is indirect. SNOpt shows first-order methods converge more slowly on centralized Neural ODEs, and FedNL and FedNS show large round savings from curvature, but only on convex static problems. Neither result transfers automatically to non-convex continuous-time models. We treat this as an open empirical question that the FedAvg/Adam baseline settles directly. If it matches our method in rounds and bits, the proposal's premise fails.

Why not just use SNOpt locally at each site and average only the weights? This is the most serious threat. It costs no extra communication, and local curvature may already remove most of the ill-conditioning. Its weakness is that each site preconditions toward its own curvature, which under temporal heterogeneity should increase client drift rather than reduce it. We run it with the same local SNOpt settings. We predict it is competitive on IID splits and degrades as heterogeneity grows. If it does not degrade, shared curvature is unnecessary and we will report that.

Why not just average one Kronecker factor pair, as distributed K-FAC does? That is the $R=1$ baseline. The cross-term argument above says exactly when it fails. The exact-Fisher diagnostic and the heterogeneity sweep measure whether that failure is large enough to matter.

Why not just send full Hessians or FedNS sketches, since the models are small? A neural CDE with tens of thousands of parameters has a Fisher on the order of $10^9$ entries per site per round. Sketching to the effective dimension, as FedNS does, is a reasonable alternative. However, FedNS relies on a square-root Hessian structure specific to generalized linear models, which a non-convex vector field does not obviously provide. We include a sketched Gauss–Newton arm on the smallest model where it is feasible.

Proposed Experiments

We use three benchmarks with controlled temporal heterogeneity. On CharacterTrajectories we create 10–20 sites, each assigned a different missingness rate between 30% and 70% and a different truncation of the observation horizon. On Speech Commands, sites subsample audio at different rates. On PhysioNet sepsis, we use site structure if the data provides it and otherwise simulate sites by stratifying patients on measurement frequency. A heterogeneity knob interpolates from IID assignment to fully regime-separated sites.

Reported metrics are rounds and uplink bits to 97% of our reproduced centralized accuracy or AUC, final accuracy, solver function evaluations per round, per-site peak memory (SNOpt adds curvature-tracking overhead that edge devices must absorb), and wall-clock time. Every result uses at least five seeds.

The decisive checks are three. The gain of $R>1$ over $R=1$ must rise monotonically with the heterogeneity knob and be near zero at IID. Regime-based grouping must beat random grouping at equal $R$. On small models, the relative error of the preconditioner against the exact Fisher must track the accuracy gap. If the first check fails, the Kronecker-sum idea is not doing the work. If only the second fails, extra terms help but temporal structure is not the reason, and we would report the method as a generic multi-term federated K-FAC.