SimTS: Rethinking Contrastive Representation Learning for Time Series Forecasting

Xiaochen Zheng, Xingyu Chen, Manuel Schürch, Amina Mollaysa, Ahmed Allam, Michael KrauthammerarXiv 2023 · OpenReview

Introduction

Contrastive learning has become a cornerstone of unsupervised representation learning in fields like computer vision and natural language processing. By teaching models to distinguish between different instances—such as identifying a cat versus a dog—these frameworks learn robust features. However, applying this "instance discrimination" approach to time series forecasting presents unique challenges that traditional methods often overlook.

The Problem with Negative Pairs

In forecasting, the primary objective is to understand the temporal evolution of a single sequence rather than distinguishing it from others. Current contrastive methods for time series often rely on the assumption that segments far apart in time should be treated as "negative pairs" (dissimilar).

The challenge of negative pairs in time series. Periodic signals cause distant segments to be similar, leading to 'false repulsion' in traditional contrastive learning.

As illustrated in Figure 1, this assumption is frequently violated by the inherent periodicity of time series data. Forcing a model to treat similar patterns occurring at different times as opposites—a phenomenon known as false repulsion—can severely degrade the quality of the learned representations. Furthermore, many existing models rely on rigid decompositions (e.g., trend vs. seasonality) that struggle with irregular or non-stationary data.

The SimTS Framework

SimTS (Simple Time Series) addresses these issues by ditching complex negative pairs and data augmentation in favor of a streamlined predictive approach. The architecture utilizes a siamese network to learn representations by predicting the future latent state from the past.

The SimTS architecture. A Siamese network uses a stop-gradient operation on the future path to learn predictive representations without needing negative pairs.

Core Components

  1. Siamese Network: Two identical encoders share weights. One processes the historical segment of the time series, while the other processes the future segment.
  2. Multi-Scale Encoder: To capture both short-term motifs and long-term global patterns, SimTS employs a bank of parallel 1D convolutions with varying kernel sizes ($2^0, 2^1, \dots, 2^m$).
  3. Predictor Head: An MLP takes the "last column" of the history's latent representation—forcing the encoder to condense all relevant historical information into a summary useful for looking ahead—and attempts to predict the future segment's representation.
  4. Stop-Gradient Operation: Borrowing a technique from SimSiam, gradients only flow through the history path. Treating the future path as a stationary target prevents the model from "collapsing" into constant outputs without requiring the computational overhead of negative pairs.

Experimental Evaluation

The authors evaluated SimTS against six public benchmark datasets, including the Electricity Transformer Temperature (ETT) subsets, Exchange-Rate, and Weather data. These benchmarks cover a wide range of sampling frequencies and stationarity levels.

Key Results

SimTS consistently outperformed both unsupervised representation learning baselines and end-to-end forecasting models:

  • Superior Accuracy: In multivariate forecasting, SimTS outperformed TS2Vec by an average of 18.8% and CoST by 11.0% in Mean Squared Error (MSE).
  • Robustness: While specialized models like CoST perform well on stationary data (e.g., Weather), they often falter on irregular data (e.g., Exchange-Rate). SimTS proved to be a consistent "all-rounder," achieving top-tier results across nearly all forecasting horizons.
  • Efficiency: By removing the need for large batches of negative pairs, SimTS offers a more direct and efficient training objective.

Conclusion

SimTS demonstrates that for time series forecasting, simplicity often trumps complexity. By rethinking the necessity of negative pairs and focusing on latent-space prediction with a stop-gradient operation, SimTS provides a robust and high-performing framework for representation learning. Future work may explore integrating this two-stage process into a seamless end-to-end pipeline and extending the model to handle irregularly sampled data.

Summary updated

BibTeX
@misc{zheng2023simts,
  author = {Xiaochen Zheng and Xingyu Chen and Manuel Schürch and Amina Mollaysa and Ahmed Allam and Michael Krauthammer},
  title = {{SimTS: Rethinking Contrastive Representation Learning for Time Series Forecasting}},
  year = {2023},
  url = {https://openreview.net/forum?id=QGmPoN4gJZd},
}
Figure