What Constitutes Good Contrastive Learning in Time-Series Forecasting?

Chiyu Zhang, Qi Yan, Lili Meng, Tristan SylvainarXiv 2023 · OpenReview

Abstract

In recent years, the introduction of self-supervised contrastive learning (SSCL) has demonstrated remarkable improvements in representation learning across various domains, including natural language processing and computer vision. By leveraging the inherent benefits of self-supervision, SSCL enables the pre-training of representation models using vast amounts of unlabeled data. Despite these advances, there remains a significant gap in understanding the impact of different SSCL strategies on time series forecasting performance, as well as the specific benefits that SSCL can bring. This paper aims to address these gaps by conducting a comprehensive analysis of the effectiveness of various training variables, including different SSCL algorithms, learning strategies, model architectures, and their interplay. Additionally, to gain deeper insights into the improvements brought about by SSCL in the context of time-series forecasting, a qualitative analysis of the empirical receptive field is performed. Through our experiments, we demonstrate that the end-to-end training of a Transformer model using the Mean Squared Error (MSE) loss and SSCL emerges as the most effective approach in time series forecasting. Notably, the incorporation of the contrastive objective enables the model to prioritize more pertinent information for forecasting, such as scale and periodic relationships. These findings contribute to a better understanding of the benefits of SSCL in time series forecasting and provide valuable insights for future research in this area.

Introduction

Self-supervised contrastive learning (SSCL) has become a cornerstone of modern representation learning in computer vision and natural language processing. By teaching models to distinguish between similar (positive) and dissimilar (negative) samples, SSCL allows for effective learning from large-scale unlabeled datasets. However, its application to time-series forecasting has remained inconsistent. This research, a collaboration between the University of British Columbia and Borealis AI, provides a systematic evaluation to determine the optimal configurations for SSCL in the forecasting domain.

The study addresses three primary gaps: the mismatch between state-of-the-art Transformer backbones and the Temporal Convolutional Networks (TCNs) typically used in SSCL research, the lack of clarity regarding "two-step" (pre-train then freeze) versus "end-to-end" training strategies, and the underlying mechanism by which contrastive learning improves forecasting performance.

Methodology

The authors evaluated a wide range of configurations across three real-world multivariate time-series benchmarks: ETT (Electricity Transformer Temperature) at hourly (ETTh1) and 15-minute (ETTm1) granularities, and ECL (Electricity Consumption Load).

Model Architectures and Learning Strategies

The study compared three backbone architectures: LSTMs, TCNs, and the Informer (a Transformer variant). Two primary learning strategies were tested:

  1. Two-Step: Pre-training an encoder with SSCL and subsequently using it as a frozen feature extractor or fine-tuning it for the forecasting task.
  2. End-to-End: Jointly training the forecaster using a combined loss function: $L = L_{MSE} + \lambda L_{SSCL}$.

Contrastive algorithms evaluated included Hierarchical Contrastive Loss (HCL), MoCo1, and MoCo2.

Empirical Receptive Field Analysis

To understand the qualitative impact of SSCL, the researchers employed Empirical Receptive Field (RF) analysis. This involves calculating the gradient of the input sequence with respect to the loss to visualize which historical data points the model prioritizes when making a prediction.

Empirical Receptive Field (RF) analysis of a TCN model trained solely with MSE loss, showing a focus on immediate history.

Results and Discussion

The experimental results indicate that the optimal setup for time-series forecasting is a Transformer backbone trained end-to-end using a combination of MSE and MoCo2-based contrastive loss.

Backbone and Strategy Performance

The Informer model trained end-to-end with MSE and MoCo2 achieved the best overall performance, with an average MSE of 0.540, outperforming TCN (0.564) and LSTM (0.800) equivalents. The study also found that end-to-end training is superior to the two-step approach. Specifically, using a frozen encoder in a two-step process led to significant performance degradation (e.g., 1.070 MSE for LSTMs), suggesting that task-specific information is lost when representations are not fine-tuned.

Optimization and Sensitivity

A key finding was the sensitivity of Transformer models to optimization strategies. While TCNs are robust to standard early stopping, Transformers often require more careful hyperparameter tuning to converge effectively. For instance, standard early stopping can lead to sub-optimal results if the model has not yet stabilized.

Loss curves for the Informer model with early stopping, illustrating the sensitivity of Transformer architectures during training.

SSCL as a Feature Filter

The RF analysis revealed that models trained only with MSE tend to focus on noisy, immediate history. In contrast, the addition of a contrastive objective acts as a "feature filter," forcing the model to prioritize more meaningful patterns, such as scale and periodic relationships (e.g., identifying cycles from 24 hours prior). This shift in focus is a primary driver of the performance gains observed when incorporating SSCL.

Conclusion

This research establishes that training a Transformer backbone end-to-end with an auxiliary MoCo2 contrastive loss is the most effective approach for time-series forecasting. While challenges remain regarding data augmentation and computational complexity, these findings provide a clear roadmap for leveraging self-supervision to improve forecasting accuracy and model interpretability.

Summary updated

Abstract

In recent years, the introduction of self-supervised contrastive learning (SSCL) has demonstrated remarkable improvements in representation learning across various domains, including natural language processing and computer vision. By leveraging the inherent benefits of self-supervision, SSCL enables the pre-training of representation models using vast amounts of unlabeled data. Despite these advances, there remains a significant gap in understanding the impact of different SSCL strategies on time series forecasting performance, as well as the specific benefits that SSCL can bring. This paper aims to address these gaps by conducting a comprehensive analysis of the effectiveness of various training variables, including different SSCL algorithms, learning strategies, model architectures, and their interplay. Additionally, to gain deeper insights into the improvements brought about by SSCL in the context of time-series forecasting, a qualitative analysis of the empirical receptive field is performed. Through our experiments, we demonstrate that the end-to-end training of a Transformer model using the Mean Squared Error (MSE) loss and SSCL emerges as the most effective approach in time series forecasting. Notably, the incorporation of the contrastive objective enables the model to prioritize more pertinent information for forecasting, such as scale and periodic relationships. These findings contribute to a better understanding of the benefits of SSCL in time series forecasting and provide valuable insights for future research in this area. Our codes are available at https://github.com/chiyuzhang94/contrastive_learning_time-series_e2e.
BibTeX
@misc{zhang2023what,
  author = {Chiyu Zhang and Qi Yan and Lili Meng and Tristan Sylvain},
  title = {{What Constitutes Good Contrastive Learning in Time-Series Forecasting?}},
  year = {2023},
  eprint = {2306.12086},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.12086},
}
Figure