Sampling-Rate Invariant Sequence Modeling

16 concepts12 papers2 proposals

Deep learning models excel at processing sequential data, but they usually share a fragile assumption: the time step between observations is fixed and immutable. When a traditional Transformer, RNN, or CNN processes a sequence, it treats the data as an ordered grid of tokens indexed by discrete steps ($t, t+1, t+2$).

This discrete coupling creates a major vulnerability. If you train a model on sensor data sampled at 100 Hz and deploy it on a device sampling at 50 Hz, or if the data contains irregular gaps due to missing measurements, the model's internal representations break down. The network confuses a change in the sampling rate with a change in the underlying dynamics. As shown by Dirk Husmeier et al. in A study on discrete-time movement models, this "time-step mismatch" introduces severe discretization errors that degrade performance when moving across diverse hardware sources.

To build models that generalize across different temporal resolutions, a growing body of research shifts the paradigm: instead of modeling discrete sequences, we must model the underlying continuous process.

Shifting to Continuous-Time Representations

The core intuition behind sampling-rate invariance is to decouple the underlying physical process from the discrete rate at which it happens to be recorded. Instead of learning a mapping between discrete tokens, the model operates on a continuous function of time, $f(t)$.

Several frameworks achieve this by changing how data is ingested and represented. For instance, Florian Heinrichs et al. proposed Functional Neural Networks in Functional Neural Networks: Shift invariant models for functional data with applications to EEG classification, which treat inputs as smooth, continuous functions rather than discrete vectors. This allows a single trained model to be applied to datasets recorded at entirely different frequencies, provided they are rescaled to a common interval.

In generative modeling, a similar philosophy applies. Elizabeth Fons et al. developed iHyperTime: Interpretable Time Series Generation with Implicit Neural Representations, which leverages implicit neural representations to map continuous time coordinates directly to data values. This allows the model to generate synthetic data at arbitrary temporal resolutions—enabling seamless upsampling and downsampling without altering the underlying neural network architecture.

The Discretization Step as a Mathematical Lever

How do we reconcile continuous-time mathematics with the discrete nature of digital computation? Modern State Space Models (SSMs) provide an elegant answer by using the discretization step size ($\Delta$) as an explicit control knob.

An SSM defines a continuous-time latent system governed by a differential equation:

$$\dot{h}(t) = Ah(t) + Bx(t)$$

To process discrete data, this continuous system must be discretized using a step size parameter, $\Delta$, which represents the actual physical time elapsed between observations. In Combining Recurrent, Convolutional, and Continuous-time Models with Linear State Space Layers, Albert Gu et al. demonstrated that this discretization step provides a rigorous mathematical foundation for handling irregularly sampled data.

The magic happens at inference time. Because the learned weights $A$ and $B$ describe the continuous system, they are completely invariant to the sampling rate. If the sampling rate changes, we do not need to retrain the model. Instead, we simply scale the step size $\Delta$ to match the new resolution.

As Albert Gu et al. demonstrated with the Structured State Space (S4) model in Efficiently Modeling Long Sequences with Structured State Spaces, adjusting $\Delta$ during inference enables zero-shot adaptation to entirely new temporal resolutions—such as varying audio frequencies or sensor rates—without specialized retraining.

Continuous System (A, B)  ───► Invariant to Sampling Rate
       │
       ├─► Discretize with Δ_high ───► High-Frequency Inference
       │
       └─► Discretize with Δ_low  ───► Low-Frequency Inference

When dealing with highly irregular or non-uniform intervals, this step size can be adjusted dynamically. Jimmy T. H. Smith et al. in Simplified State Space Layers for Sequence Modeling showed that varying $\Delta$ at each individual timestep allows the same parameters to handle non-uniform intervals seamlessly. To push this further, Aoran Wang et al. in Structural Inference of Dynamical Systems with Conjoined State Space Models parameterized $\Delta$ as a learnable function of the input data itself, allowing the model to dynamically adapt its internal temporal representation to match highly irregular trajectories.

Functional Decoders and Sampling-Rate Equivariance

While adjusting $\Delta$ handles the input side, generating forecasts or outputs that are invariant to the sampling rate requires a corresponding strategy on the decoding side. A prominent solution is to interpret the model's outputs not as discrete future points, but as the coefficients of a continuous functional basis.

This approach is exemplified by the FlowState architecture developed by Lars Graf et al. in FlowState: Sampling Rate Invariant Time Series Forecasting. By pairing an SSM encoder with a Functional Basis Decoder (FBD), the model outputs coefficients that define a continuous curve over time. As the authors detail in FlowState: Sampling-Rate Invariant Time Series Foundation Model with Dynamic Forecasting Horizons, this continuous representation allows the model to naturally adapt to varying sampling rates and dynamic forecasting horizons without retraining.

In FlowState: Sampling-Rate‑Equivariant Time‑Series Forecasting, the authors formalize this property as sampling-rate equivariance. By treating time series as continuous signals, the model learns the underlying physics of the data in a normalized space. This prevents the severe error spikes that traditional discrete models experience when encountering uncommon or unseen sampling intervals, enabling robust zero-shot generalization across a continuous range of temporal scales.

Why It Works: Learning the Underlying Physics

Ultimately, sampling-rate invariance works because it forces neural networks to learn the true continuous laws governing a system rather than the artifacts of its discretization.

When a model is designed to be uniform across different sampling rates, its reconstruction and inference accuracy remain stable even as the sampling interval approaches zero, converging gracefully to a continuous-time limit (José Bento et al. in Learning Networks of Stochastic Differential Equations).

This mathematical consistency is vital for cross-domain transferability. For example, in autonomous driving, trajectory data collected from different cities or vehicles often features varying frequencies. By representing trajectories as continuous curves via Neural Stochastic Differential Equations (NSDEs), Daehee Park et al. demonstrated in Improving Transferability for Cross-Domain Trajectory Prediction via Neural Stochastic Differential Equation that models trained on high-frequency data can be applied directly to low-frequency environments without losing accuracy.

By anchoring the model's internal representations to physical time rather than discrete sequence indices, continuous-time architectures achieve a level of robustness and zero-shot adaptability that traditional discrete sequence models cannot match.

Key Foundational Work

A study on discrete-time movement models (2019) — contributed 1 concept elements to this direction.
Learning Networks of Stochastic Differential Equations (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sampling_rate_invariance_d216cb45,
  title = {sampling-rate invariance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sampling-rate-invariance/rdr_d216cb45b2604056b687d18ffdb46005},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_sampling_rate_invariance_d216cb45_v1,
  title = {sampling-rate invariance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sampling-rate-invariance/rdr_d216cb45b2604056b687d18ffdb46005/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake