TLNets: Transformation Learning Networks for long-range time-series prediction

Wei Wang, Yang Liu, Hao SunarXiv 2023 · DOI · arXiv

Abstract

TLNets (Transformation Learning Networks) introduce a novel framework for long-range time-series prediction by leveraging mathematical transformations like Fourier Transforms and SVD to achieve "global" receptive fields. By fusing these global views with local convolutional features, TLNets achieve state-of-the-art accuracy on long-horizon forecasting while providing a more interpretable and efficient alternative to complex Transformer-based models.

Introduction

Predicting the future of a time series, such as electricity usage or stock prices, becomes exponentially harder as the prediction horizon increases. Traditional models face a significant trade-off: statistical models like ARIMA are robust but struggle with complex multivariate data, while CNNs are excellent at local patterns but require deep architectures to achieve a sufficient receptive field. Transformers capture long-range dependencies but are computationally expensive and often see performance degradation as horizons grow.

TLNets address this by introducing Receptive Field Learning (RFL). The core idea is that instead of stacking layers to increase a model's "view," we should utilize mathematical transformations that naturally possess a global view of the data from the outset.

Methods

TLNets are constructed from four fundamental "Transformation Blocks":

  1. Fourier Transform (FT) Block: Maps data to the frequency domain to learn global periodic patterns.
  2. Singular Value Decomposition (SVD) Block: Captures correlations between variables and learns spatiotemporal dynamics through an eigen/modal perspective.
  3. Sparse Matrix Block: A trainable matrix designed to learn both local and global information simultaneously.
  4. Convolution (Conv) Block: Captures standard local, neighboring dependencies.

By combining these blocks, the authors developed four specific architectures: FT-Matrix, FT-SVD, FT-Conv, and Conv-SVD. The overarching strategy is to project features into a properly defined domain (like the Fourier or orthonormal domain) where learning is more efficient for long-range tasks.

Architecture of TLNets showing the integration of Fourier and SVD blocks to achieve global receptive fields.

Key Insights

  • Global for Free: While a CNN requires many layers to see a 720-step horizon, a single Fourier Transform "sees" the entire sequence at once, providing global context in shallow, efficient layers.
  • Mathematical Equivalence: The paper provides a rigorous proof showing that 1D Convolutions are a specific form of sparse matrix multiplication, bridging deep learning with classical linear algebra.
  • Frequency Bottleneck: Shallow CNN layers are mathematically biased toward low-frequency information. The FT block bypasses this, allowing the model to capture high-frequency signals more effectively.
  • Orthogonality: Using SVD helps maintain orthogonality in model weights, leading to better convergence and more stable training compared to standard dense layers.

Key Results

TLNets were evaluated on standard benchmarks including ETT (Electricity Transformer Temperature), Traffic, Electricity, Weather, and Exchange-Rate datasets. The models consistently outperformed state-of-the-art baselines like Autoformer, FEDformer, and Informer, particularly in long-range scenarios.

Performance comparison of TLNets variants against state-of-the-art models like Autoformer across MSE, MAE, and Correlation metrics.

On the ETTm1 dataset, TLNets (specifically FT-SVD) showed significantly lower Mean Squared Error (MSE) than Transformers as the prediction horizon increased to 720 steps. Furthermore, ablation studies confirmed that the hybrid approach is necessary; models using only SVD or only FT performed worse than the combined TLNet architectures, proving that balancing local and global RFL is essential for performance.

Conclusion and Future Work

TLNets provide a powerful, mathematically grounded alternative to Transformers for long-horizon forecasting. While highly effective, the framework currently relies on manual selection of the appropriate transformation (e.g., Fourier vs. SVD). Future work may involve automating this selection process and optimizing the memory requirements of the FT-Matrix variant for even larger datasets. Additionally, exploring the application of the RFL framework to computer vision and natural language processing remains a promising direction.

Summary updated

Abstract

Time series prediction is a prevalent issue across various disciplines, such as meteorology, traffic surveillance, investment, and energy production and consumption. Many statistical and machine-learning strategies have been developed to tackle this problem. However, these approaches either lack explainability or exhibit less satisfactory performance when the prediction horizon increases. To this end, we propose a novel plan for the designing of networks' architecture based on transformations, possessing the potential to achieve an enhanced receptive field in learning which brings benefits to fuse features across scales. In this context, we introduce four different transformation mechanisms as bases to construct the learning model including Fourier Transform (FT), Singular Value Decomposition (SVD), matrix multiplication and Conv block. Hence, we develop four learning models based on the above building blocks, namely, FT-Matrix, FT-SVD, FT-Conv, and Conv-SVD. Note that the FT and SVD blocks are capable of learning global information, while the Conv blocks focus on learning local information. The matrix block is sparsely designed to learn both global and local information simultaneously. The above Transformation Learning Networks (TLNets) have been extensively tested and compared with multiple baseline models based on several real-world datasets and showed clear potential in long-range time-series forecasting.
BibTeX
@misc{wang2023tlnets,
  author = {Wei Wang and Yang Liu and Hao Sun},
  title = {{TLNets: Transformation Learning Networks for long-range time-series prediction}},
  year = {2023},
  eprint = {2305.15770},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.15770},
}
Figure