TL;DR
TS2Vec is a universal framework that learns robust, multi-scale representations of time series data at any semantic level—from individual timestamps to full sequences. By utilizing hierarchical contrastive learning over augmented context views, a single trained TS2Vec model achieves state-of-the-art performance across three distinct downstream tasks: classification, forecasting, and anomaly detection.
What Problem Does This Paper Address?
Most existing unsupervised time series representation methods suffer from three core limitations:
- Coarse Granularity: They primarily focus on instance-level representations (summarizing an entire time series segment). While useful for classification, this coarse graining fails on fine-grained tasks like forecasting or anomaly detection, which require localized, timestamp-level predictions.
- Lack of Multi-Scale Awareness: They rarely capture multi-scale contextual information, missing out on scale-invariant features that are crucial for generalization.
- Flawed Inductive Biases: Many methods borrow data augmentation strategies directly from computer vision or natural language processing (e.g., random cropping or transformations like scaling and permutation). However, time series data distributions often shift over time due to anomalies or level shifts. Forcing a model to be invariant to these changes can destroy critical semantic information.
TS2Vec tackles these challenges by learning representations at the timestamp level, which can then be flexibly aggregated (via max pooling) to represent any arbitrary sub-sequence or full instance.
Methods
The TS2Vec architecture consists of an input projection layer, a timestamp masking module, and a dilated CNN backbone with residual blocks to capture long-range dependencies.
The framework relies on two primary innovations:
- Contextual Consistency: Instead of forcing the model to be invariant to aggressive transformations, TS2Vec requires that the representation of the same timestamp remains consistent across two different augmented context views of the same sequence. Contexts are generated using a combination of Random Cropping (sampling overlapping segments) and Timestamp Masking.
- Hierarchical Contrasting: The model optimizes a dual contrastive loss—temporal contrastive loss (distinguishing the same timestamp from others in the same sequence) and instance-wise contrastive loss (distinguishing the sequence from other sequences in the batch). This loss is computed hierarchically by recursively applying max pooling along the time axis, forcing the encoder to capture features at multiple granularities.
Datasets
The authors evaluated TS2Vec across a diverse array of standard benchmarks:
- Time Series Classification: 125 UCR univariate datasets and 29 UEA multivariate datasets.
- Time Series Forecasting: Three Electricity Transformer Temperature (ETT) datasets and the Electricity dataset.
- Anomaly Detection: The Yahoo benchmark dataset (367 hourly sampled series) and the KPI competition dataset (minutely sampled curves from Internet companies).
Key Insights and Ideas
- Latent-Space Masking over Raw Masking: In time series, data values can be unbounded, making it impossible to define a universal "mask token" (like
[MASK]in NLP). Masking raw values with0can confuse the model if0is a valid data point. TS2Vec solves this by masking the latent vectors after the input projection layer. - Cropping Prevents Representation Collapse: Fully convolutional networks can implicitly encode absolute positional information. Without random cropping, a model performing temporal contrastive learning can cheat by simply learning absolute time embeddings (e.g., outputting a one-hot vector for index $t$), leading to representation collapse. Random cropping forces the network to learn position-agnostic contextual features.
- Robustness to Missing Data: Because of the combination of timestamp masking (which trains the model to infer missing context) and hierarchical contrasting (which captures long-range trends), TS2Vec is exceptionally robust to missing data, maintaining high classification accuracy even when 50% of the data points are missing.
Key Results
TS2Vec establishes a new state-of-the-art (SOTA) across all three target domains:
- Classification: Outperforms all previous unsupervised methods, improving average accuracy by 2.4% on UCR and 3.0% on UEA datasets.
- Forecasting: A simple linear regression model trained on top of frozen TS2Vec representations outperformed complex, specialized SOTA forecasting models (like Informer and N-BEATS), achieving a 32.6% reduction in average MSE for univariate forecasting.
- Anomaly Detection: In a normal setting, TS2Vec improves the $F_1$ score by 18.2% on the Yahoo dataset and 5.5% on the KPI dataset compared to previous unsupervised baselines. It also demonstrates strong transferability in cold-start scenarios.
- Efficiency: TS2Vec provides the shortest training time among the compared representation learning methods, taking under an hour for the entire UCR/UEA archives on a single GPU.
Limitations and Future Work
While TS2Vec demonstrates impressive universality, the authors note that the number of negative samples in a batch (controlled by batch size) can have a notable, dataset-dependent impact on performance. Adjusting this hyperparameter away from the default may be necessary for optimal results on specific individual datasets. The authors suggest that extending this hierarchical contrastive framework to other non-time-series domains remains a promising direction for future research.
Summary updated