Learning from Polar Representation: An Extreme-Adaptive Model for Long-Term Time Series Forecasting

Yanhong Li, Jack Xu, David C. AnastasiuAAAI 2024 · DOI

TL;DR

This paper introduces DAN, a novel model for long-term streamflow forecasting that excels at predicting extreme events. By learning "polar representations" of data (both extreme and normal values) and using a unique oversampling strategy, DAN significantly outperforms existing methods on challenging hydrological datasets.

What Problem Does This Paper Address?

Accurate long-term forecasting of streamflow is critical for managing water resources, preventing floods, and mitigating droughts. However, existing time series forecasting models struggle with two major challenges: capturing long-range dependencies and accurately predicting rare but impactful extreme events (like flash floods or severe droughts). These extreme events, while infrequent, have disproportionately large consequences, making their accurate prediction crucial. Traditional statistical methods and even many advanced deep learning models often fail when faced with the imbalanced nature of extreme data and the need to predict far into the future.

Methods

The core of DAN lies in its innovative approach to representation learning and handling imbalanced data.

  • RepGen (Representation Generation): This stage uses three parallel encoder-decoder blocks to learn different "polar" representations:
  • Far points (ŷf): Captures extreme values.
  • Near points (ŷn): Captures normal values.
  • Indicator (ŷi): Learns a representation to specifically flag potential extreme events. This separation ensures that extreme event features aren't diluted by the majority of normal data.

  • RepMerg (Representation Merging): This stage combines the learned polar representations and the indicator to refine predictions. It uses a gate control vector mechanism to dynamically weigh the importance of extreme vs. normal predictions.

  • Distance-weighted Multi-Loss: DAN employs a custom loss function that uses different weights to emphasize accuracy for points far from the mean (extreme values) and points close to the mean (normal values). This encourages the model to learn robust representations for both.
  • Kruskal-Wallis Sampling: To address the imbalanced nature of extreme events, the paper introduces a non-parametric sampling policy. It uses the Kruskal-Wallis test to identify time segments likely to contain significant changes (potential extremes) and oversamples them during training.
  • GMM Indicator: If no external indicator data (like rainfall) is available, DAN can generate its own indicator using a Gaussian Mixture Model (GMM) based on the univariate time series itself.

Datasets

The study evaluated DAN on four real-life hydrologic streamflow datasets from Santa Clara County, CA: Ross, Saratoga, UpperPen, and SFC. Each dataset included streamflow and associated rainfall data. The training and validation sets spanned from January 1988 to August 2021, with inference predicting streamflow every 4 hours for the next 3 days (h=288 time steps).

Key Insights and Ideas

  • Polarity Matters: Separating the learning of "extreme" (far) and "normal" (near) data representations is crucial for accurately forecasting time series with imbalanced extreme events.
  • Indicator is Key: A dedicated indicator representation, refined through CNNs, significantly aids in predicting extreme events.
  • Oversampling for Extremes: A non-parametric sampling strategy (Kruskal-Wallis) can effectively boost a model's ability to learn from rare extreme events without completely discarding normal data.
  • Adaptive Regularization: Distance-weighted loss functions can act as a powerful regularizer, forcing the model to pay more attention to critical data points.

Key Results

  • Superior Performance: DAN significantly outperformed seven state-of-the-art baselines (including FEDFormer, Informer, NLinear, DLinear, NEC+, LSTM-Atten, and N-BEATS) on all four benchmark datasets for both multivariate and univariate forecasting.
  • Significant RMSE Reduction: In multivariate forecasting, DAN achieved an average relative RMSE reduction of 19% compared to the second-best model, with a notable 30% improvement on the UpperPen dataset.
  • Handling Extremes: Qualitative analysis (Figure 5) showed DAN better capturing the peaks of extreme events compared to other models, which tended to either under- or over-predict during these critical periods.
  • Ablation Studies: Experiments confirmed the importance of each key component: the refine layer, RepMerg, and regularization loss all contributed to DAN's superior performance.

Limitations and Future Work

  • Extensible Framework: The paper notes that the optimal configuration of the "repeatable stacks" (E, D, R) in RepGen and RepMerg can vary across datasets, suggesting that further research into automatically determining these configurations could be beneficial.
  • Oversampling Policy Tuning: While effective, the Kruskal-Wallis sampling policy involves tuning parameters (ε and p). Future work could explore more automated or adaptive ways to set these parameters.
  • Generalizability: While tested on four hydrological datasets, exploring DAN's performance on other types of time series with extreme events (e.g., financial data, sensor readings) would be valuable.

Summary updated

BibTeX
@inproceedings{li2024learning,
  author = {Yanhong Li and Jack Xu and David C. Anastasiu},
  title = {{Learning from Polar Representation: An Extreme-Adaptive Model for Long-Term Time Series Forecasting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i1.27768},
}
Figure