Breaking feature collapse in self-supervised time series encoders

Özgün Turgut, Daniel RueckertarXiv 2026 · arXiv

TL;DR

Current general-purpose time series encoders suffer from "feature collapse," where increasing model size fails to yield better performance because the learned representations become redundant. By adding feature-space supervision—specifically self-distillation and entropy-maximizing regularization—the authors introduce OTISv2, a model that finally scales monotonically and matches the performance of models 43 times its size.

What Problem Does This Paper Address?

In natural language processing and computer vision, "scaling laws" are the gold standard: bigger models trained on more data almost always perform better. However, time series analysis has hit a wall. As shown in Figure 1, existing state-of-the-art encoders like UniTS and MOMENT saturate quickly; a massive 308M parameter model barely outperforms a tiny 7.1M parameter one.

Figure 1: General-purpose time series encoders saturate at scale. Average accuracy over 23 UEA and 120 UCR datasets. Across three general-purpose encoder families, gains through parameter scaling sat- urate. OTIS-7.1M nearly matches the 43× larger MOMENT-308M.

The authors trace this failure to feature collapse. Most time series models use Masked Data Modeling (MDM), where the encoder learns by reconstructing hidden patches of input. The problem is that MDM only supervises the input space. Without constraints on the feature space, the encoder's output tokens become highly redundant, effectively "collapsing" into a low-dimensional subspace. This makes the extra capacity of larger models useless.

Methods

To break this collapse, the authors developed OTISv2. It builds on the original OTIS architecture but introduces two critical feature-space objectives to complement the standard reconstruction loss:

  1. Self-Distillation: Inspired by DINO, the model uses a student-teacher framework. A student encoder tries to match the output distribution of an exponential-moving-average (EMA) teacher. This forces the encoder to learn semantically consistent features across different masked views of the same data.
  2. KoLeo Regularizer: This loss function rewards "spreading" features out across the unit hypersphere. By penalizing near-duplicate features across different samples, it directly counteracts the tendency of tokens to cluster together (collapse).

Architecturally, OTISv2 also moves away from domain-specific embeddings. It adopts Axial Rotary Position Embeddings (RoPE) to handle varying sequence lengths and variate counts and adds register tokens to store global information, freeing up patch tokens to focus on local waveform structures.

Datasets

The authors curated a massive, diverse pre-training corpus:

  • Scale: 756,267 samples totaling 45.2 billion time points.
  • Diversity: 11 domains including healthcare (ECG, EEG), audio, engineering, natural sciences (weather, temperature), and finance.
  • Evaluation: The model was tested on 143 datasets from the UEA (multivariate) and UCR (univariate) archives, covering tasks like 5-shot classification and linear probing.

Key Insights and Ideas

  • Input-Space Supervision is Insufficient: Reconstructing a signal doesn't guarantee high-quality features. If the model can reconstruct the input using only a few dimensions of its feature space, it will, leading to redundancy.
  • The "RankMe" Metric: The authors use RankMe (a measure of effective rank) to quantify feature quality. They show that as standard models grow, their RankMe stays flat or drops, while OTISv2’s RankMe increases, indicating it is actually using its expanded capacity.
  • Scaling is Finally Unlocked: By supervising the feature space, the authors demonstrate that time series encoders can finally follow the scaling trends seen in other modalities. OTISv2 scales monotonically from 7.1M to 116M parameters.

Key Results

OTISv2 represents a significant leap in efficiency and performance:

  • Efficiency: OTISv2-7.1M matches or surpasses the performance of MOMENT-308M (a model 43x larger) across the 143-dataset benchmark.
  • Feature Diversity: As shown in Figure 2, OTISv2 significantly reduces patch-token cosine similarity and increases the effective rank (RankMe) compared to its predecessor.
  • Superior Downstream Performance: On the UEA 5-shot accuracy benchmark, OTISv2-7.1M achieves 70.12%, outperforming all other general-purpose encoders in its class.
Figure 2: Feature quality comparison. OTISv2 (blue) maintains lower token similarity and higher rank than the original OTIS (red) as training progresses.

Limitations and Future Work

  • Generative Tasks: The evaluation focused primarily on discriminative tasks (classification). The authors note that the impact of these feature-space objectives on generative tasks like forecasting remains to be fully explored.
  • Compute Overhead: The self-distillation teacher increases the memory required for training by about 2.4x, though this does not affect inference speed.
  • Short Sequences: The fixed patch width ($p=24$) limits performance on very short univariate series (e.g., 96 time points), where the encoder only sees a handful of tokens.
  • Noise Sensitivity: OTISv2 appears more sensitive to additive noise and spikes than the original OTIS, likely because its features are more specialized and less redundant.

Summary updated

Abstract

General-purpose time series encoders promise a single backbone that generalises across tasks and domains; however, they currently do not scale. We trace this down to a critical point of failure: under the predominant masked data modelling supervision, the encoder's output tokens partially collapse. This is the natural consequence of supervising encoders only in the input space, which leaves no constraint against redundant token dimensions in the feature space. We introduce OTISv2, which suppresses the collapse by complementing the reconstruction objective in input space with two objectives in the feature space: a self-distillation loss against an exponential-moving-average teacher, and a KoLeo regulariser. Under matched training, the resulting encoder reduces the cosine similarity and lifts the effective rank of its output tokens relative to its predecessor, and it scales monotonically from 7.1 M to 116 M parameters where its predecessor declines. It matches or surpasses existing general-purpose time series encoders across 143 uni- and multi-variate datasets. To foster further research, we release our code and pre-trained weights.
BibTeX
@misc{turgut2026breaking,
  author = {Özgün Turgut and Daniel Rueckert},
  title = {{Breaking feature collapse in self-supervised time series encoders}},
  year = {2026},
  eprint = {2611.20981},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20981},
}
Figure