Empirical-Bayes Random Effects for Multi-Outcome Neural ODEs

Hospitals, disease registries and cohort studies record the same people again and again: several lab values, scans and test scores per person, taken at irregular times, with some people seen a dozen times and others only twice. Neural ordinary differential equations (Neural ODEs) are networks that learn a smooth rule for how a hidden state changes over time, so they can be queried at any time point. They suit this kind of data, but they personalize in an ad hoc way. Statisticians have handled people seen only twice for decades with hierarchical (mixed-effects) models. These models split each person's parameters into a population-wide part and a person-specific deviation, then pull that deviation toward zero by an amount set by how much data the person has. We propose to put that shrinkage inside a Neural ODE, computed by the classical statistical estimator rather than guessed by a neural encoder. We then treat the several measured outcomes as tasks that share one person-specific deviation vector. Its learned covariance decides how much one outcome should inform another.

The two-sentence version is this. Each person's dynamics are a shared learned vector field plus a low-dimensional random-effect vector, and that vector is estimated per person by linearizing the ODE solution and applying the closed-form best-linear-unbiased-predictor (BLUP) shrinkage of hierarchical linear models. Because all outcomes of a person share this vector through a learned covariance, a densely measured outcome can sharpen forecasts of a sparsely measured one. The covariance also learns to switch off that sharing when outcomes are unrelated, which gives multi-task learning a statistical answer to negative transfer, the case where training on one task hurts another.

Core Claim

Existing continuous-time models give each subject its own trajectory in one of two ways. Some infer it with a learned encoder. The Latent ODE of Latent Ordinary Differential Equations for Irregularly-Sampled Time Series infers a per-subject starting state with a backward ODE-RNN, and Neural ODE Processes infers a global dynamics vector from context points. Others regress subject parameters from metadata, as CNODE does for time shift and speed in Conditional Neural ODE for Longitudinal Parkinson's Disease Progression Forecasting. In all of these, how far a subject is pulled toward the population depends on what the encoder happened to learn, not on how many observations that subject has. We claim that for repeated-measures data with very few visits per subject, an explicit Gaussian random effect estimated by linearized empirical Bayes gives better-calibrated and more accurate forecasts in the sparse regime. We further claim that sharing this random effect across outcomes, with a full learned covariance, transfers information between outcomes without the negative transfer that a shared-encoder multi-head model suffers.

Concretely, subject $i$'s latent state $z_i(t)$ follows

$$\frac{dz_i}{dt} = f_\theta(z_i, t) + G_\theta(z_i, t)\, b_i, \qquad b_i \sim \mathcal{N}(0, \Sigma), \qquad z_i(0) = z_0 + C\, b_i ,$$

where $f_\theta$ is the shared (fixed-effect) vector field, $G_\theta$ is a learned matrix-valued network mapping the $q$-dimensional random effect $b_i$ (with $q$ around 4–16) into a velocity perturbation, and $C$ lets part of $b_i$ act as a random intercept on the starting state. Each outcome $k$ is a task with its own readout, $y_{ikj} = h_k(z_i(t_{ikj})) + \varepsilon$ with noise variance $\sigma_k^2$, observed on its own irregular schedule. The vector $S_i(t) = \partial z_i(t)/\partial b_i$ is computed by forward sensitivity equations (that is, $q$ extra ODE columns). Linearizing the stacked observations $y_i$ around a current estimate gives $y_i \approx m_i + A_i b_i + \varepsilon_i$, with $A_i$ built from the readout Jacobians times $S_i$. The random effect can then be integrated out exactly:

$$y_i \sim \mathcal{N}\!\left(m_i,\; A_i \Sigma A_i^\top + R_i\right), \qquad \hat b_i = \Sigma A_i^\top \left(A_i \Sigma A_i^\top + R_i\right)^{-1}(y_i - m_i),$$

where $R_i$ is the diagonal noise covariance. The second expression is the hierarchical-model BLUP. With few observations, $A_i \Sigma A_i^\top$ is small relative to $R_i$ and $\hat b_i$ shrinks toward zero. With many observations the subject keeps its own deviation. We iterate the linearization around $\hat b_i$ a few times, which is the "first-order conditional" approximation used in classical nonlinear mixed-effects software. We train $\theta$, $\Sigma$ and the $\sigma_k$ by maximizing the resulting Laplace marginal likelihood. The off-diagonal structure of $\Sigma$, projected through each task's readout, is what moves information between tasks for a given subject.

Relation to Prior Work

The closest work is Mixed Effects Neural ODE (ME-NODE). It already writes the vector field as a network times a vector of fixed and random effects, $\dot z = \Gamma(z) w^i$, and trains it inside a VAE with an ABC-derived ELBO on Rotating MNIST and ADNI MRI. It supplies our parameterization. We differ in two specific ways. First, we replace its encoder-based variational inference over $w^i$ with closed-form linearized empirical Bayes, so shrinkage is driven by each subject's data count. Second, we model several asynchronously observed outcomes through a learned full covariance $\Sigma$, where ME-NODE models a single outcome with a simple prior. Its authors flag richer random-effect priors as future work.

Modeling longitudinal imaging biomarkers with parametric Bayesian multi-task learning motivates the proposal. Treating subjects as tasks coupled by empirical-Bayes shrinkage roughly halved ADNI prediction MAE relative to uncoupled fits. But it uses static parametric curves and couples subjects, not outcomes. We keep its empirical-Bayes logic and move it into a learned continuous-time vector field.

Multi-Task Learning via Time-Aware Neural ODE (NORMAL) and RotoGrad attack negative transfer differently. NORMAL places tasks at learned positions along an ODE over network depth, not physical time. RotoGrad rotates the shared feature space per task to align gradients. Both treat samples as independent and act on the optimization or architecture. Our mechanism instead acts on the data model: the shared parameters are coupled only to the degree $\Sigma$ supports. These papers constrain our evaluation, because they are the natural task-conflict baselines.

MeNets showed that fixed-plus-random-effect decomposition helps deep networks on subject-clustered gaze data, but statically and for one task. The original Neural Ordinary Differential Equations paper supplies the solver and sensitivity machinery.

Research Program

The work has three parts.

The first part builds the estimator. It needs efficient batched forward-sensitivity computation for $q$ random-effect directions, a numerically stable BLUP via Cholesky factors of the per-subject $n_i \times n_i$ covariance (with $n_i$ the subject's total observation count), and a Laplace objective whose gradient passes through the solver. When an outcome is non-Gaussian, such as in-hospital mortality, we treat it as a further task whose likelihood is approximated at $\hat b_i$, as in generalized mixed models.

The second part tests whether shrinkage is calibrated. We measure, as a function of each subject's observation count, the error and the coverage of predictive intervals. These intervals come for free from the Laplace posterior over $b_i$ propagated through $S_i$.

The third part studies cross-task transfer. We read the learned $\Sigma$, mapped into each outcome's readout space, as an estimate of which outcomes co-vary within people. We test whether this estimate predicts when sharing helps.

Baselines

The naive reference points are two pooling extremes. The first is a population-only Neural ODE with $b_i \equiv 0$ (complete pooling). The second is a per-subject fit that conditions the vector field on a learned subject embedding with no prior (no pooling). Any useful method must beat both, and the no-pooling model shows how badly unregularized personalization fails at two visits.

The strongest encoder-based competitor is a multi-head Latent ODE from Rubanova et al., with one readout per outcome and a shared ODE-RNN encoder. On PhysioNet 2012 that paper reports ODE-RNN patient-outcome MSE of 2.361 against 3.215 for the next baseline. It is the strongest because it already personalizes through an amortized subject-specific starting state.

The strongest method overall on irregular clinical series in the file is mTAN from Multi-Time Attention Networks. It reports PhysioNet 2012 mortality AUC of 0.858 and Human Activity accuracy of 0.910, and trains 85–100 times faster than ODE models.

The closest structural competitor is ME-NODE trained with its own variational objective and given the same multiple readouts. On Rotating MNIST it reports interpolation MSE of 0.0027 against 0.0640 for ODE2VAE.

For task conflict we add RotoGrad and NORMAL-style learned task positions on top of the multi-head Latent ODE. For classical statistics we fit a multivariate linear mixed model and the Marquand-style coupled Bayesian model on ADNI. For ADNI and PPMI there is no reported result found on Lacuna for exact MAE or MSE values, so we rerun every baseline ourselves.

All methods get the same splits, the same latent dimension where applicable, the same hyperparameter search budget (an equal number of trials over learning rate, width and latent size), and the same wall-clock cap. We report wall-clock time separately.

Two ablations isolate the key ingredients. The first keeps our exact architecture but infers $b_i$ with an amortized encoder instead of the BLUP. This tests whether closed-form shrinkage is the active ingredient. The second restricts $\Sigma$ to be block-diagonal by outcome. This tests whether cross-task covariance is what produces transfer.

Beating the field means two things. First, lower forecast error and closer-to-nominal 90% interval coverage on subjects with three or fewer observations of the target outcome than both the encoder ablation and the multi-head Latent ODE, with no loss on dense subjects. Second, on PhysioNet, mortality AUC at least matching mTAN's 0.858 while also winning on extrapolation beyond the observed window.

Why not just…?

Why not just use a multi-head Latent ODE, whose encoder already personalizes each subject? This is the most serious threat. A well-trained encoder with a standard normal prior does some shrinkage. Nothing in the file shows that it shrinks by the right amount for subjects with two visits. The encoder ablation answers the question directly: the same model, the same random-effect dimension, with only the inference changed. If the ablation matches us in the sparse stratum, the proposal reduces to ME-NODE with more heads, and we would report that.

Why not just use mTAN, which is far faster and already strong on PhysioNet? On interpolation and classification within the observed window mTAN may win, and we expect it might. Rubanova et al. and Norcliffe et al. argue that ODE models extrapolate better because attention-style models fall back toward mean values outside the training window. Our forecasting splits hide the last 30% of each subject's time span, and our cross-task split hides one outcome entirely for a test window. These test exactly the regimes where a subject-specific dynamics model should matter. If mTAN wins there too, the ODE machinery is not justified for these data.

Why not just fit a classical multivariate mixed model with parametric time curves? On ADNI, with two to four visits per person and slow near-linear change, a linear mixed model may be very hard to beat. Marquand et al. show that coupled parametric curves already give large gains there. We include it and expect our method to match it on ADNI. We expect our method to win only where dynamics are non-linear and more densely sampled (PhysioNet, synthetic oscillators), and we will state which regime each dataset falls into.

Why not just add gradient homogenization to a shared multi-task ODE? RotoGrad and NORMAL fix conflicting gradients over the shared weights. Our hypothesis is that in repeated-measures data much of the apparent conflict is a modeling error: outcomes are forced to share subject variation that they do not share. The synthetic experiment below varies true cross-outcome correlation and checks whether RotoGrad on a shared encoder closes the gap that a learned block-diagonal $\Sigma$ would close.

Proposed Experiments

The first experiment is synthetic and decisive. We simulate subjects from a known mixed-effects ODE (damped oscillators and a two-compartment progression model) with three outcomes. We vary the true within-subject correlation of their random effects from −0.8 through 0 to 0.8, and the visits per subject from 2 to 20. The predictions are as follows. The learned $\Sigma$ should recover the true correlations. Gains on a sparsely observed outcome should grow with correlation magnitude. At zero correlation our method should show no degradation relative to single-task fits, whereas the shared-encoder multi-head Latent ODE should degrade. Error and coverage curves against visit count should show our advantage concentrated at 2–4 visits and vanishing at 20.

The second experiment uses PhysioNet Challenge 2012. Variable groups (vitals, blood chemistry, blood gases) are tasks, and mortality is an extra task predicted from $\hat b_i$ and the latent state. We report interpolation MSE, extrapolation MSE on held-out late windows, and mortality AUC against mTAN's 0.858.

The third experiment uses ADNI and PPMI. Regional volumes or cortical thickness and cognitive scores are tasks with different schedules. PPMI's average visit gap of 1.11 years and frequent two- or three-visit subjects make it a natural sparse stress test. Following CNODE, we evaluate MSE and $R^2$ on each subject's last visit, stratified by number of prior visits. Cross-task transfer is tested by hiding the cognitive score at the final visit and predicting it from imaging alone.

As a sanity check against published numbers, we also report Variable Rotating MNIST interpolation against ME-NODE's 0.0027.

Across all experiments we report per-epoch time against the multi-head Latent ODE, ME-NODE and mTAN. The $q$ sensitivity columns and per-subject Cholesky solves make the cost explicit. We will also report whether a random-effect dimension of $q \le 8$ captures most of the benefit, which decides whether the method is practical at scale.