Abstract
TLNets (Transformation Learning Networks) introduce a novel framework for long-range time-series prediction by leveraging mathematical transformations like Fourier Transforms and SVD to achieve "global" receptive fields. By fusing these global views with local convolutional features, TLNets achieve state-of-the-art accuracy on long-horizon forecasting while providing a more interpretable and efficient alternative to complex Transformer-based models.
Introduction
Predicting the future of a time series, such as electricity usage or stock prices, becomes exponentially harder as the prediction horizon increases. Traditional models face a significant trade-off: statistical models like ARIMA are robust but struggle with complex multivariate data, while CNNs are excellent at local patterns but require deep architectures to achieve a sufficient receptive field. Transformers capture long-range dependencies but are computationally expensive and often see performance degradation as horizons grow.
TLNets address this by introducing Receptive Field Learning (RFL). The core idea is that instead of stacking layers to increase a model's "view," we should utilize mathematical transformations that naturally possess a global view of the data from the outset.
Methods
TLNets are constructed from four fundamental "Transformation Blocks":
- Fourier Transform (FT) Block: Maps data to the frequency domain to learn global periodic patterns.
- Singular Value Decomposition (SVD) Block: Captures correlations between variables and learns spatiotemporal dynamics through an eigen/modal perspective.
- Sparse Matrix Block: A trainable matrix designed to learn both local and global information simultaneously.
- Convolution (Conv) Block: Captures standard local, neighboring dependencies.
By combining these blocks, the authors developed four specific architectures: FT-Matrix, FT-SVD, FT-Conv, and Conv-SVD. The overarching strategy is to project features into a properly defined domain (like the Fourier or orthonormal domain) where learning is more efficient for long-range tasks.
Key Insights
- Global for Free: While a CNN requires many layers to see a 720-step horizon, a single Fourier Transform "sees" the entire sequence at once, providing global context in shallow, efficient layers.
- Mathematical Equivalence: The paper provides a rigorous proof showing that 1D Convolutions are a specific form of sparse matrix multiplication, bridging deep learning with classical linear algebra.
- Frequency Bottleneck: Shallow CNN layers are mathematically biased toward low-frequency information. The FT block bypasses this, allowing the model to capture high-frequency signals more effectively.
- Orthogonality: Using SVD helps maintain orthogonality in model weights, leading to better convergence and more stable training compared to standard dense layers.
Key Results
TLNets were evaluated on standard benchmarks including ETT (Electricity Transformer Temperature), Traffic, Electricity, Weather, and Exchange-Rate datasets. The models consistently outperformed state-of-the-art baselines like Autoformer, FEDformer, and Informer, particularly in long-range scenarios.
On the ETTm1 dataset, TLNets (specifically FT-SVD) showed significantly lower Mean Squared Error (MSE) than Transformers as the prediction horizon increased to 720 steps. Furthermore, ablation studies confirmed that the hybrid approach is necessary; models using only SVD or only FT performed worse than the combined TLNet architectures, proving that balancing local and global RFL is essential for performance.
Conclusion and Future Work
TLNets provide a powerful, mathematically grounded alternative to Transformers for long-horizon forecasting. While highly effective, the framework currently relies on manual selection of the appropriate transformation (e.g., Fourier vs. SVD). Future work may involve automating this selection process and optimizing the memory requirements of the FT-Matrix variant for even larger datasets. Additionally, exploring the application of the RFL framework to computer vision and natural language processing remains a promising direction.
Summary updated