TL;DR
Continuous-time models like Neural Controlled Differential Equations (NCDEs) typically require interpolating irregular data into continuous paths, which can introduce artificial biases and heavy computational overhead. This paper proves that explicit interpolation is entirely unnecessary if the data embedding is continuous and injective ("faithful"). The authors introduce a novel, interpolation-free embedding for Log-NCDEs that directly summarizes irregular observations into interval log-signatures, achieving up to a 3,000× speedup on long sequences while remaining highly robust to missing data.
What Problem Does This Paper Address?
Real-world time-series data—such as electronic health records or financial logs—is notoriously messy. Observations arrive at irregular intervals, channels are sampled asynchronously, and data is often only partially observed. Standard deep learning architectures (like Transformers or State-Space Models) naturally expect data on a regular grid.
To bridge this gap, continuous-time models embed discrete observations into continuous paths. However, the standard practice relies on imputation or interpolation (e.g., cubic splines) to "fill in the blanks" before the model even sees the data. This design choice forces the model to consume invented, unobserved dynamics and artificial values. The authors of this paper tackle a fundamental question: How can we represent irregular, asynchronous data in continuous time without making arbitrary guesses about unobserved behavior?
Methods
The paper introduces two core contributions to solve this problem:
- The Universality Transfer Lemma: The authors mathematically prove that a downstream model's universal approximation capabilities on an input space transfer directly back to the original data space, provided the data embedding is continuous and injective. They term this a faithful embedding. Crucially, the embedding does not need to be smooth.
- Direct Interval Log-Signatures: Guided by this theory, they design an embedding specifically for Log-NCDEs. Instead of constructing an interpolated path, they convert discrete observations directly into time-stamped increments and observation counts. These increments are treated as local log-signature contributions within a truncated free Lie algebra.
Using Chen's identity and a parallel associative scan, these local factors are efficiently composed over arbitrary query intervals to form interval log-signatures ($\Phi_{\alpha, \beta}$) without any intermediate interpolation. This approach is integrated into a structure-preserving variant of Structured Linear CDEs, termed Log-SLiCE, enabling massive parallelization.
Datasets
The authors evaluated their approach across several challenging setups:
- EigenWorms: A benchmark from the UEA multivariate time-series classification archive featuring exceptionally long sequences (17,984 observations per series).
- Synthetic Sinusoidal Coupled Data: A custom two-channel system driven by a shared latent phase, evaluated under four distinct sampling regimes (synchronous regular, synchronous irregular, asynchronous irregular, and asynchronous sparse).
- Brownian Rough Paths: A 4-dimensional Brownian stream on a fine regular grid (2,048 steps) used to drive a Stratonovich linear controlled system, testing the model's capacity to exploit higher-order geometric information (Lévy area).
- UEA Archive (6 Datasets): A suite of classification datasets used to evaluate model robustness under severe input dropping (up to 95%).
Key Insights and Ideas
- Stop Guessing, Start Embedding: Traditional preprocessing steps try to guess what happens between observations. This paper argues that guessing should be handled by the modeling layer, not the representation layer. The representation should strictly preserve what was actually observed.
- Injectivity is All You Need: For a downstream universal model to succeed, the embedding doesn't need to look pretty or smooth; it just needs to keep distinct data objects distinguishable (injective) and ensure nearby data remains nearby (continuous).
- Observation Counts Matter: By explicitly tracking an "observation count" coordinate, the embedding successfully distinguishes between a channel that repeats its previous value and a channel that simply wasn't measured at that timestamp.
- Native Higher-Order Information: Because this framework bypasses path construction, it can natively ingest precomputed higher-order geometric features (like Lévy areas), which standard interpolation completely discards.
Key Results
- Massive Computational Speedups: On the long-sequence EigenWorms dataset, combining the Log-ODE method with a SLiCE architecture (Log-SLiCE) reduced the training time per step by a factor of nearly 3,000× compared to standard NCDEs. This compressed a 3-day training run down to just 1.5 minutes while boosting test accuracy by 11.1 percentage points.
- Extreme Robustness to Missing Data: Under a severe 95% input dropping regime on the UEA datasets, the block-diagonal Log-SLiCE (BD-SLiCE) maintained an average accuracy of 62.5% (down only slightly from 64.0% with no dropping), significantly outperforming classical recurrent baselines like Mamba and S5.
- Exploiting Rough Path Geometry: In the Brownian motion experiment, incorporating level-2 log-signatures (area information) yielded substantially lower mean squared error (MSE) than using level-1 increments alone, demonstrating the framework's unique ability to leverage higher-order data structure.
Limitations and Future Work
- Exponential Coordinate Growth: The size of the truncated tensor and log-signature representations scales exponentially with the truncation depth $N$ and the number of channels. Enforcing exact injectivity via auxiliary channels can become computationally expensive for very high-dimensional streams.
- Invariance Trade-offs: Intentionally omitting count coordinates or physical time introduces mathematical invariances. While this simplifies the model, it can discard critical information in timing-sensitive or event-driven tasks.
- Empirical Scope: The empirical validation was primarily restricted to classification and synthetic controlled systems. Exploring how this faithful representation performs in large-scale generative modeling or forecasting remains an open direction.
Summary updated