TL;DR
Current general-purpose time series encoders suffer from "feature collapse," where increasing model size fails to yield better performance because the learned representations become redundant. By adding feature-space supervision—specifically self-distillation and entropy-maximizing regularization—the authors introduce OTISv2, a model that finally scales monotonically and matches the performance of models 43 times its size.
What Problem Does This Paper Address?
In natural language processing and computer vision, "scaling laws" are the gold standard: bigger models trained on more data almost always perform better. However, time series analysis has hit a wall. As shown in Figure 1, existing state-of-the-art encoders like UniTS and MOMENT saturate quickly; a massive 308M parameter model barely outperforms a tiny 7.1M parameter one.
The authors trace this failure to feature collapse. Most time series models use Masked Data Modeling (MDM), where the encoder learns by reconstructing hidden patches of input. The problem is that MDM only supervises the input space. Without constraints on the feature space, the encoder's output tokens become highly redundant, effectively "collapsing" into a low-dimensional subspace. This makes the extra capacity of larger models useless.
Methods
To break this collapse, the authors developed OTISv2. It builds on the original OTIS architecture but introduces two critical feature-space objectives to complement the standard reconstruction loss:
- Self-Distillation: Inspired by DINO, the model uses a student-teacher framework. A student encoder tries to match the output distribution of an exponential-moving-average (EMA) teacher. This forces the encoder to learn semantically consistent features across different masked views of the same data.
- KoLeo Regularizer: This loss function rewards "spreading" features out across the unit hypersphere. By penalizing near-duplicate features across different samples, it directly counteracts the tendency of tokens to cluster together (collapse).
Architecturally, OTISv2 also moves away from domain-specific embeddings. It adopts Axial Rotary Position Embeddings (RoPE) to handle varying sequence lengths and variate counts and adds register tokens to store global information, freeing up patch tokens to focus on local waveform structures.
Datasets
The authors curated a massive, diverse pre-training corpus:
- Scale: 756,267 samples totaling 45.2 billion time points.
- Diversity: 11 domains including healthcare (ECG, EEG), audio, engineering, natural sciences (weather, temperature), and finance.
- Evaluation: The model was tested on 143 datasets from the UEA (multivariate) and UCR (univariate) archives, covering tasks like 5-shot classification and linear probing.
Key Insights and Ideas
- Input-Space Supervision is Insufficient: Reconstructing a signal doesn't guarantee high-quality features. If the model can reconstruct the input using only a few dimensions of its feature space, it will, leading to redundancy.
- The "RankMe" Metric: The authors use RankMe (a measure of effective rank) to quantify feature quality. They show that as standard models grow, their RankMe stays flat or drops, while OTISv2’s RankMe increases, indicating it is actually using its expanded capacity.
- Scaling is Finally Unlocked: By supervising the feature space, the authors demonstrate that time series encoders can finally follow the scaling trends seen in other modalities. OTISv2 scales monotonically from 7.1M to 116M parameters.
Key Results
OTISv2 represents a significant leap in efficiency and performance:
- Efficiency: OTISv2-7.1M matches or surpasses the performance of MOMENT-308M (a model 43x larger) across the 143-dataset benchmark.
- Feature Diversity: As shown in Figure 2, OTISv2 significantly reduces patch-token cosine similarity and increases the effective rank (RankMe) compared to its predecessor.
- Superior Downstream Performance: On the UEA 5-shot accuracy benchmark, OTISv2-7.1M achieves 70.12%, outperforming all other general-purpose encoders in its class.
Limitations and Future Work
- Generative Tasks: The evaluation focused primarily on discriminative tasks (classification). The authors note that the impact of these feature-space objectives on generative tasks like forecasting remains to be fully explored.
- Compute Overhead: The self-distillation teacher increases the memory required for training by about 2.4x, though this does not affect inference speed.
- Short Sequences: The fixed patch width ($p=24$) limits performance on very short univariate series (e.g., 96 time points), where the encoder only sees a handful of tokens.
- Noise Sensitivity: OTISv2 appears more sensitive to additive noise and spikes than the original OTIS, likely because its features are more specialized and less redundant.
Summary updated