Respecting Time Series Properties Makes Deep Time Series Forecasting Perfect

Li Shen, Yuning Wei, Yangzhu WangarXiv 2022 · OpenReview

Abstract

How to handle time features shall be the core question of any time series forecasting model. Ironically, it is often ignored or misunderstood by deep-learning based models, even those baselines which are state-of-the-art. This behavior makes them inefficient, untenable, and unstable. In this paper, we rigorously analyze three prevalent but deficient deep time series forecasting mechanisms from the view of time series properties, including normalization methods, multivariate forecasting, and input sequence length. Corresponding corollaries and solutions are given on both empirical and theoretical bases. We thereby propose a novel time series forecasting network, RTNet, on the basis of aforementioned analysis. It is general enough to be combined with both supervised and self-supervised forecasting formats. Thanks to the core idea of respecting time series properties, RTNet shows obviously superior forecasting performances compared with dozens of other SOTA time series forecasting baselines in three real-world benchmark datasets. By and large, it even occupies less time complexity and memory usage while acquiring better forecasting accuracy.

Introduction

The research landscape for deep time series forecasting is booming, with architectures like Transformers, CNNs, and GNNs dominating leaderboards. However, many state-of-the-art (SOTA) models blindly adopt deep learning mechanisms—such as Batch Normalization, naive multivariate concatenation, and excessively long input sequences—that fundamentally contradict the nature of time series data.

Real-world time series are non-stationary, causal, and lack scale invariance. When models ignore these properties, they become inefficient and prone to overfitting. This paper identifies which standard deep learning practices harm forecasting and proposes mathematically sound alternatives through a new architecture: RTNet (Respecting Time Network).

Methodology: The RTNet Architecture

RTNet is designed to bridge the gap between deep learning power and time series statistical requirements through four primary innovations.

Preserving Scale with Weight Normalization

Standard techniques like Batch Normalization (BN) and Layer Normalization (LN) re-parameterize hidden units, assuming that scale disparity can be ignored. Because time series are not scale-invariant, this destroys critical information. RTNet exclusively uses Weight Normalization (WN), which re-parameterizes weights instead of hidden units, preserving the data's statistical characteristics and causality.

Multivariate Forecasting via Cos-Relation Matrix

Blindly feeding all available variables into a model often hurts performance if some variables are independent of the target. RTNet introduces a Cos-Relation Matrix that calculates the cosine similarity between variables. It weights the input sequences so that irrelevant variables are suppressed, preventing them from adding noise to the prediction.

The Cos-Relation Matrix mechanism for weighting multivariate inputs based on variable similarity.

Causal Pyramid Network (CPN)

Many models maintain a constant input sequence length throughout the network. RTNet borrows a pyramid structure from Computer Vision (similar to ResNet). As the network gets deeper, it progressively halves the sequence length while doubling the feature channels. This extracts hierarchical, multi-scale temporal features with significantly less computation than traditional flat architectures.

The RTBlock (a) and the overall RTNet architecture (b), illustrating the pyramid structure and feature mapping.

Decoupled Time Embedding

Instead of mixing time features (like "hour-of-day") directly into the input sequence, RTNet processes them through a separate, independent network. This prevents temporal metadata from interfering with the raw signal processing in the early layers.

Experimental Evaluation

RTNet was evaluated on three standard real-world benchmark datasets: ETT (Electricity Transformer Temperature), WTH (Weather), and ECL (Electricity Consuming Load).

Performance and Efficiency

RTNet consistently outperformed dozens of SOTA baselines across both univariate and multivariate tasks. Notably, it was the first model to achieve a Mean Squared Error (MSE) below 0.03 for univariate forecasting on ETTh1 at a prediction length of 24.

Beyond accuracy, RTNet is highly efficient. In training comparisons, RTNet required only 11 seconds per run in its contrastive format, compared to over 1 minute for TS2Vec and over 4 minutes for SCINet, demonstrating the computational benefits of the Causal Pyramid Network.

Key Insights

  • The "Long-Term Dependency" Myth: Increasing input sequence length to capture long-term dependencies often leads to sparse attention scores where the model only looks at adjacent elements. Shorter, appropriate input lengths prevent overfitting.
  • Normalization Sensitivity: Simply removing Layer Normalization from models like Informer drastically changes performance. Normalization must be chosen based on data properties, not just standard DL practice.
  • Variable Filtering: Adding independent variables increases model complexity (BIC score) without improving likelihood. Filtering variables via the Cos-Relation Matrix is a necessity for stable forecasting.

Conclusion

RTNet demonstrates that respecting the fundamental properties of time series data—scale, causality, and variable relevance—is more effective than simply increasing model depth or sequence length. By prioritizing these properties, RTNet achieves state-of-the-art accuracy with superior computational efficiency.

Summary updated

Abstract

How to handle time features shall be the core question of any time series forecasting model. Ironically, it is often ignored or misunderstood by deep-learning based models, even those baselines which are state-of-the-art. This behavior makes their inefficient, untenable and unstable. In this paper, we rigorously analyze three prevalent but deficient/unfounded deep time series forecasting mechanisms or methods from the view of time series properties, including normalization methods, multivariate forecasting and input sequence length. Corresponding corollaries and solutions are given on both empirical and theoretical basis. We thereby propose a novel time series forecasting network, i.e. RTNet, on the basis of aforementioned analysis. It is general enough to be combined with both supervised and self-supervised forecasting format. Thanks to the core idea of respecting time series properties, no matter in which forecasting format, RTNet shows obviously superior forecasting performances compared with dozens of other SOTA time series forecasting baselines in three real-world benchmark datasets. By and large, it even occupies less time complexity and memory usage while acquiring better forecasting accuracy. The source code is available at https://github.com/OrigamiSL/RTNet.
BibTeX
@misc{shen2022respecting,
  author = {Li Shen and Yuning Wei and Yangzhu Wang},
  title = {{Respecting Time Series Properties Makes Deep Time Series Forecasting Perfect}},
  year = {2022},
  eprint = {2207.10941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.10941},
}
Figure