Median-of-Means in Clock Time for Irregular, Heavy-Tailed Time Series

Many sequences that models are trained on, such as sensor streams, ICU vitals, and market prices, have three awkward properties at once. Measurements arrive at uneven times. Each value depends on the recent past. Every so often the stream contains a large, sudden excursion, such as a sensor fault or a price shock, that is far bigger than Gaussian noise would allow. These excursions do not arrive one point at a time. Because the process has memory, a shock persists over a stretch of time. Because observation is often triggered by events, that stretch tends to be sampled more densely than calm periods. A training loss that averages over observations therefore gives these bursts outsized weight, and a continuous-time model fitted to it can be dragged far from the true dynamics.

We propose a robust training and evaluation objective that addresses the three properties together. It is a median-of-means estimator whose blocks are defined by elapsed physical time rather than by sample count. Median-of-means is a robust estimate of an average: split the data into groups, average each group, and take the median of the group averages. A single wild value can spoil only one group. Our version cuts the trajectory into equal-duration windows separated by unused buffer windows. The buffers make the windows nearly independent despite temporal memory. Equal duration ensures that a densely sampled burst lands in one or two windows instead of many. We train continuous-time sequence models by minimizing the median window loss. We also prove a high-probability bound for the estimator under mixing, meaning memory that fades with time separation, measured in continuous time. The central prediction is falsifiable. The objective should beat per-point robust losses such as Huber exactly when heavy-tailed events are clustered in time. It should beat sample-count median-of-means exactly when sampling density rises during those events.

Core Claim

Existing robustness tools for time series each drop one of the three properties. On Empirical Risk Minimization with Dependent and Heavy-Tailed Data proves that Huber loss restores fast rates under β-mixing, a standard way of saying that dependence between two points decays as their separation grows. It works on a uniform integer time grid and explicitly lists median-of-means for dependent data as open. Lasso guarantees for β-mixing heavy-tailed time series likewise stays on uniform grids with linear models. The classical median-of-means guarantees of Hsu et al. and Minsker assume independent samples. The blocking arguments for dependent data, from Meir through Mohri et al. and Fu et al., define blocks by index length. Under irregular sampling, the same index length can span five minutes or five hours. The dependence that a buffer is supposed to remove is therefore uncontrolled.

On the modeling side, continuous-time architectures handle irregular timestamps but train on squared or Gaussian likelihood losses. The file's overview flags that such models can diverge under heavy-tailed noise spikes. This group includes Latent ODEs, GRU-ODE-Bayes, ContiFormer and DualDynamics. The last of these uses "robust" to mean tolerance to missingness, not to outliers. The closest robust continuous-time model is EDICT. It predicts an evidential Normal-Inverse-Wishart distribution and down-weights incoming points that exceed a predicted-variance threshold at inference time. That is a per-point, Gaussian-calibrated filter with no guarantee, and it does not change what the model is trained to fit.

Our claim is that the unit of robustness in irregular dependent data should be a window of clock time, not an observation or a fixed count of observations. Choosing that unit correctly gives both a provable estimator and a measurable training advantage in the burst-clustered regime.

The Estimator and What We Can Prove

The setup is as follows. Take a trajectory observed at irregular times $t_1 < \dots < t_n$ in $[0, T]$, with per-observation losses $\ell_i$ under a model. Choose a window duration $L$ and a buffer duration $b$. Let $B_1, \dots, B_K$ be the kept windows, each of duration $L$, separated by buffers of duration $b$, so $K \approx T/(L+b)$. The clock-time median-of-means risk is

$$\hat R_{\mathrm{CT}} = \operatorname{median}_{j \le K} \; \frac{1}{|B_j|} \sum_{t_i \in B_j} \ell_i ,$$

where $|B_j|$ is the number of observations falling in window $j$. Windows with no observations are dropped.

The theorem we will prove combines three steps. First, a coupling argument (the standard blocking technique) replaces the kept windows by independent copies. This costs at most $K\beta(b)$, where $\beta(b)$ is the continuous-time mixing coefficient at time separation $b$. Second, Chebyshev's inequality makes each window mean accurate with constant probability. This needs only a finite variance $\sigma_L^2$ for the window mean, taken jointly over the noise and the sampling process. Third, a binomial tail bound controls the chance that more than half the windows are inaccurate. Together these give, with probability at least $1 - e^{-cK} - K\beta(b)$,

$$|\hat R_{\mathrm{CT}} - R| \le C\,\sigma_L ,$$

where $R$ is the stationary risk and $c, C$ are absolute constants.

The substantive work lies in $\sigma_L$. We will bound it in terms of the observation-rate process and the tail index of the noise. Two consequences follow from this bound. First, equal-duration windows keep $\sigma_L$ comparable across windows even when sampling rates vary tenfold. Second, a burst of duration shorter than $L+b$ can corrupt at most two windows, so it cannot move the median once $K$ is moderate. Index-based windows lack both properties. During a densely sampled burst they allocate many windows to the same event.

We deliberately do not claim a generalization bound for non-convex Neural ODEs. The file's own gaps note that the stability theory of Stability beyond Bounded Differences is i.i.d.-only. Extending it to continuous-time mixing is a separate project. The theory here certifies the objective and the evaluation estimator. Training behavior is tested empirically. The same result also gives a practical rule: choose $b$ from an estimated autocorrelation time, and choose $K$ on the order of $\log(1/\delta)$ for a target failure probability $\delta$.

Research Program

We will build the objective as a drop-in loss for three continuous-time backbones: GRU-ODE-Bayes, ContiFormer, and a Latent ODE. Training takes gradients through the median window. A soft-median variant, which weights windows by closeness to the median, is also tested for gradient smoothness. For datasets made of many short independent trajectories, such as ICU patients, the windows are formed within each trajectory and pooled. The median then runs over the pooled windows.

We will also construct a benchmark variant, because none exists that combines hard dynamics with heavy-tailed, time-clustered corruption (Gap 4 in the file). Physiome-ODE provides 50 biological ODE systems with known clean trajectories and a difficulty score, Joint Gradient Deviation, that correlates with best-model error (Spearman 0.77). It uses Gaussian noise and random dropping. We will replace the noise with three controlled components. The first is Pareto-tailed shocks with tail index $p \in \{1.5, 2.5, 4\}$. The second is shock persistence governed by an autoregressive decay with half-life $h$. The third is event-triggered sampling, whose rate rises by a factor $\rho$ during shocks. The knobs $h$ and $\rho$ correspond directly to the two predicted mechanisms. Because clean trajectories are known, error is always measured against the true dynamics, not the corrupted observations. We will also evaluate on real data with naturally irregular sampling and probable heavy tails, namely USHCN climate, the Google stock series, PhysioNet 2012 and MIMIC-III. For the real data we report both the natural setting and a shock-injected setting.

Baselines

The naive reference points are two. The first is each backbone trained with its standard squared or likelihood loss. The second is the constant last-observation forecast. Physiome-ODE showed that this forecast often matches Neural ODEs on PhysioNet 2012 and MIMIC-III. Any gain we report on those datasets must first clear it.

The strongest existing forecasting methods on the target benchmark are LinODEnet and GraFITi. Physiome-ODE reports that LinODEnet is best on 29 of 50 datasets and GraFITi on 10 of 50, under Gaussian noise. We will run both unchanged and also with our objective swapped in. We found no reported result on Lacuna for any method on Pareto-perturbed Physiome-ODE, so all numbers in that setting will be generated here.

The strongest robust continuous-time competitor is EDICT with its variance-guided reweighting. On MIMIC mortality under injected noise it reports AUROC near 0.80 while baselines fall toward 0.50. On PhysioNet mortality interpolation, DualDynamics reports test MSE 3.114 at 90% observation, and we use that as the clean reference number. Beating EDICT means exceeding its AUROC under matched noise injection on MIMIC. On Physiome-ODE with shocks, it means lower clean-trajectory MSE and lower 99th-percentile error across series and seeds. We report the high-quantile error because, as High-probability minimax lower bounds stresses, average error hides exactly the tail failures robustness targets.

Two ablations isolate the key ingredient. The first is Huber loss on each backbone, with its threshold tuned. The second is sample-count median-of-means, with the same number of windows $K$ and matched buffers in index units. Every method gets the same backbone, data splits, training compute, and a 30-trial hyperparameter search per method and dataset. The Huber threshold, the EDICT variance threshold, and our $L$ and $b$ all count within that budget.

Why not just…?

Why not just use Huber loss on a continuous-time model? This is the real threat. On Empirical Risk Minimization with Dependent and Heavy-Tailed Data already shows Huber gives fast rates under mixing, and it is one line of code. Our argument is specific rather than general. Huber caps each residual, but a persistent shock produces many consecutive capped residuals pointing the same way, and their sum still biases the fit. If shocks are isolated ($h \to 0$), we expect Huber to match us, and we will report that honestly. The burst-duration sweep settles the question: the proposal is justified only if the gap over Huber grows with $h$.

Why not just run ordinary median-of-means on index blocks with buffers? This is the obvious transfer of i.i.d. median-of-means plus classical blocking. It should work whenever sampling is roughly uniform in time. It should fail when observation frequency rises during shocks, as in clinical monitoring, because the burst then occupies many index blocks. The $\rho$ sweep tests exactly this. If index blocks match clock-time windows even at large $\rho$, the time-based construction adds nothing, and we will say so.

Why not just filter outliers at inference, as EDICT does, or forward-fill onto a grid? Inference-time reweighting in EDICT protects the latent state from incoming spikes. The model's dynamics, however, were still learned from contaminated training targets. Its threshold is also calibrated under a Gaussian-conjugate model, which the authors note struggles with asymmetric heavy tails. We will test EDICT's reweighting combined with our objective as well as each alone, to see whether the two effects are complementary. Forward-filling onto a grid erases the event-triggered sampling signal, as the file notes for early sepsis prediction. It also copies each shock forward until the next observation, which should worsen error in exactly the clustered regime. We include it only as part of the discrete-model reference.

Decisive Checks

Four results would confirm or kill the proposal. First, on the shocked Physiome-ODE variant, the advantage over tuned Huber must be near zero at $h = 0$ and increase monotonically with shock half-life. Second, the advantage over index median-of-means must be near zero at $\rho = 1$ and increase with $\rho$. Third, on clean Gaussian Physiome-ODE, our objective should lose at most a small efficiency margin to squared loss. A large loss would make it impractical. Fourth, the empirical failure rate of the clock-time estimator, used as a risk estimate, should track the $e^{-cK} + K\beta(b)$ form. Specifically, it should drop sharply once the buffer $b$ exceeds the estimated autocorrelation time, and it should not improve further as $b$ grows beyond that point. If the first two checks come out flat, the conclusion is that per-point robustness suffices for irregular time series, and that result is itself worth reporting.