TL;DR
Rather than chopping time series into consecutive time chunks for next-token prediction, Narratives of Time Series (NoTS) re-imagines time series modeling as predicting across levels of functional abstraction. By applying degradation operators (such as local smoothing and low-pass filtering), NoTS generates a coarse-to-fine sequence of temporal functions and trains an autoregressive transformer to progressively reconstruct the original signal. This theoretically bypasses approximation bottlenecks inherent in standard patching methods and yields consistent performance improvements across classification, imputation, and anomaly detection.
What Problem Does This Paper Address?
Mainstream time-series transformers borrow heavily from natural language processing: they slice continuous temporal signals into consecutive time intervals (patches) and predict future intervals autoregressively. However, this time-slicing paradigm has fundamental flaws:
- Breaks non-local properties: Chunking disrupts global features like multi-scale trends and long-term periodicity.
- Theoretical approximation limits: The authors demonstrate that sampling common signal operators (such as differentiation) over raw time slices creates discontinuous sequence-to-sequence functions that standard transformers fail to approximate under the universal approximation framework.
While some prior work compensates with Fourier neural operators or Koopman operators, these usually require fixed mathematical bases or specialized coding blocks that do not transfer easily across diverse domains.
Methods
NoTS reformulates time series as continuous functions $g(t)$ sampled over time, modeling them through a next-function prediction framework:
Raw Signal g(t) ───► Apply Degradations [d₁(·), d₂(·), ...] ───► Coarse-to-Fine Functions [S₁, S₂, ..., Sₖ]
│
Encoder & Latent Tokens
│
Target Sₖ (Fine) ◄─── Decoder ◄─── AR Transformer (Causal Attention) ◄────┘
- Functional Degradation Operators: Instead of arbitrary masking or fixed bases, NoTS constructs data-dependent sequences using two smoothing operators of varying intensity:
- Local smoothing: Moving average filters with decreasing window sizes.
- Global smoothing: Sinc low-pass filters with increasing frequency cutoffs.
- Latent Autoregressive Modeling: Tokenizers (1D-ResNet encoder/decoder) map each degraded version $S_k$ into latent token cohorts $R_k$. A causal transformer with group-wise masking predicts token representations of finer resolutions from coarser ones.
- Objective Function: Combines an autoregressive reconstruction loss (predicting $S_{k+1}$ from previous states) with a latent consistency term that reconstructs the raw signal directly from its uncorrupted latent representation, preventing representation drift.
- Context-Aware Adaptation: For downstream transfer, NoTS uses a lightweight prompt-tuning strategy:
- Channel adaptors: Linear projection layers and dataset-specific channel embeddings.
- Task adaptors: Deep visual prompt tuning tokens prepended to transformer layers plus task-specific heads, updating fewer than 1% of total parameters.
Datasets
The authors evaluated NoTS across synthetic and real-world benchmarks:
- Synthetic Benchmarks:
- Fractional Brownian Motion (fBm): 20,000 trajectories of length 1,024 generated via Cholesky decomposition across diverse Hurst indices ($H \in (0, 1)$).
- Superposition of Autocorrelated Sinusoids: 50,000 synthesized multi-frequency series generated via five autoregressive frequency processes.
- Real-World Benchmarks (22 datasets total):
- Classification: 9 univariate datasets from the UCR archive and 5 multivariate datasets from the UEA archive.
- Anomaly Detection: SMD, MSL, SWaT, and PSM benchmarks.
- Imputation: 4 datasets from the ETDataset family (ETTm1, ETTm2, ETTh1, ETTh2).
Key Insights and Ideas
- Resolution Over Time: Slicing signals along the resolution/complexity axis creates a natural "narrative" from low-frequency baseline shapes to high-frequency variations, analogous to how humans decompose a signal.
- Theoretical Expressiveness: The paper proves that standard transformers attempting to approximate a differential operator over raw time steps can suffer an error bounded below by $\sqrt{T}$. Feeding multi-scale functional abstractions solves this discontinuity, restoring universal approximation guarantees.
- Smoothing Beats Noise: Unlike standard diffusion models that degrade data with Gaussian noise, time series benefit significantly more from deterministic smoothing. Ablations revealed that Gaussian corruption severely degraded performance compared to local/global filtering.
Key Results
- Synthetic Feature Regression: NoTS-lw outperformed VQ-VAE, standard MAE, frequency-aware MAE (FAMAE), and next-period predictors, achieving up to a 37.8% relative error reduction on Hurst index regression and an average 26% improvement across fBm features.
- Real-World Multitask Transfer:
- In parameter-frozen adaptation (<1% parameters trained), NoTS-lw beat next-period prediction and masked autoencoders, achieving the lowest average error rate (18.51 vs. 24.39 for next-period AR).
- Under full fine-tuning, NoTS-lw attained an average error rate of 15.10, outperforming SimMTM (16.14) and bioFAME (16.05).
- Plug-and-Play Architectural Boost: Adding NoTS pre-training to popular architectures consistently reduced downstream error:
- PatchTST: Average error rate dropped from 21.78 to 18.33.
- iTransformer: Average error rate dropped from 16.07 to 15.70.
- Model Scaling: A parameter sweep (127k to 2.1M parameters) demonstrated that test reconstruction loss steadily decreases following a power-law trend.
Limitations and Future Work
- Model and Data Scale: The largest evaluated model contained 2.1M parameters; validating scaling laws on multi-billion-parameter foundation models across web-scale data remains open.
- Theoretical Ties to Diffusion: While connection to "cold diffusion" (deterministic transformations) is noted, an analytical bridging framework between spectral autoregression and continuous diffusion formulations is needed.
- Stochastic and Irregular Signals: Further exploration is required to test how well NoTS handles irregularly sampled series or non-deterministic jump processes (e.g., via rough path theory or neural SDEs).
Summary updated