RWKV-TS: Beyond Traditional Recurrent Neural Network for Time Series Tasks

Haowen Hou, F. Richard YuarXiv 2024 · DOI · arXiv

Introduction

Time series analysis has long been dominated by Recurrent Neural Networks (RNNs) like LSTMs and GRUs. However, these models have recently struggled to keep pace with Transformers and CNNs due to three primary "fatal flaws": vanishing gradients in long sequences, sequential bottlenecks that prevent parallel training, and error accumulation during long-horizon forecasting.

RWKV-TS addresses these challenges by adapting the "Receptance Weighted Key Value" (RWKV) architecture—originally developed for Large Language Models—to the time series domain. It bridges the gap between the parallelizable training of Transformers and the efficient, constant-memory inference of RNNs, achieving state-of-the-art performance with linear $O(L)$ complexity.

The RWKV-TS Architecture

The core innovation of RWKV-TS is its ability to maintain the recursive nature of an RNN while adopting the parallel processing capabilities of a Transformer.

The RWKV-TS architecture featuring Time Mixing and Channel Mixing modules.

Key technical components include:

  • Linear Attention (WKV Operator): Unlike the quadratic $O(L^2)$ attention in Transformers, the WKV operator mirrors self-attention but operates with $O(L)$ time and space complexity.
  • Token Shift: A "time mixing" mechanism that linearly interpolates the current input $x_t$ with the previous input $x_{t-1}$, ensuring effective information flow across time steps.
  • Dual-Mode Execution: The model can run in Parallel Mode for fast training on GPUs and Recurrent Mode for memory-efficient inference.
  • Patching & Normalization: To handle time series data effectively, the model uses Instance Normalization to combat distribution shifts and Patching to group time steps into tokens, focusing on local semantic patterns.

Experimental Results

The authors evaluated RWKV-TS across various tasks, including long-term and short-term forecasting, classification, anomaly detection, and imputation.

Performance and Efficiency

In long-term forecasting, RWKV-TS achieved a 12.58% average MSE reduction compared to TimesNet. It matches the accuracy of leading Transformer models like PatchTST while being significantly more efficient.

Comparison of parameters, training time, and inference time across different models.

As shown in the data above, RWKV-TS-768 is approximately 3x faster in both training and inference compared to FEDformer, despite having a comparable parameter count. This efficiency makes it ideal for real-time industrial monitoring and edge computing.

Robustness and Versatility

RWKV-TS also demonstrated strong few-shot learning capabilities. When trained on only 10% of the data, it showed a 28.95% MSE reduction over TimesNet, suggesting it learns more robust representations. In classification tasks, it outperformed all other RNN-based models and most Transformer baselines with an average accuracy of 73.1%.

Discussion and Future Work

The success of RWKV-TS suggests a "renaissance" for RNN-like architectures. By replacing traditional gating with decay-based linear attention, the model can capture dependencies in sequences exceeding 4,000 steps—far beyond the 100-step limit of traditional LSTMs.

However, a current limitation is in data imputation. Because RWKV-TS is inherently unidirectional (a "look-back" model), it lacks the "future" context that bidirectional Transformers use to fill gaps. Future work aims to develop a bidirectional version (Bi-RWKV-TS) and explore scaling the model into a large-scale "Foundation Model" for time series.

Summary updated

BibTeX
@misc{hou2024rwkv,
  author = {Haowen Hou and F. Richard Yu},
  title = {{RWKV-TS: Beyond Traditional Recurrent Neural Network for Time Series Tasks}},
  year = {2024},
  eprint = {2401.09093},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.09093},
}
Figure