LeRet: Language-Empowered Retentive Network for Time Series Forecasting

Qihe Huang, Zhengyang Zhou, Kuo Yang

Abstract

Time series forecasting (TSF) is essential for numerous real-world applications, yet traditional models often struggle with generalization and data scarcity. While Large Language Models (LLMs) offer a potential solution, their computational cost and the modality gap between text and numerical data remain significant barriers. We introduce LeRet, a novel framework that integrates the causal modeling efficiency of Retentive Networks (RetNet) with the domain knowledge of frozen LLMs. By extracting linguistic features from time series descriptions and aligning them with numerical data through a cross-attention mechanism, LeRet achieves state-of-the-art performance with extreme parameter efficiency. Our evaluations show that LeRet outperforms existing models in 57 out of 64 long-term forecasting settings and excels in zero-shot scenarios.

Introduction

The challenge of time series forecasting lies in capturing complex temporal dependencies and generalizing across diverse datasets. Traditional deep learning approaches like CNNs and Transformers often require large amounts of task-specific data. Conversely, LLM-based models, while powerful, are often too large for practical deployment and struggle to directly process continuous numerical sequences. LeRet addresses these challenges by using language as a qualitative prior, allowing the model to leverage "common sense" knowledge about temporal patterns (e.g., trends, noise) extracted from a frozen LLM.

Methodology

LeRet's architecture is designed to bridge the gap between numerical time series and linguistic knowledge through three main components:

  1. Causal Dependence Learner (CDL): Unlike bidirectional Transformers, LeRet employs a Retentive Network (RetNet) backbone to capture the inherent causal evolution of time. It incorporates a patch-level pre-training task to refine its understanding of temporal segments.
  2. TS-Related Language Knowledge Extractor (TRLE): This module uses a Language Description Protocol to characterize time series based on trend, periodicity, stability, and noise. These descriptions are processed by a frozen LLM (e.g., LLaMA) to generate high-level text embeddings.
  3. Language-TS Modality Integrator (TLMI): A two-stage cross-attention mechanism aligns the extracted linguistic embeddings with numerical features, "empowering" the numerical data with qualitative context before final prediction.
The four-stage architecture of LeRet, illustrating the integration of causal dependence learning and language-modality alignment.

Experimental Results

We evaluated LeRet on a comprehensive suite of benchmarks, including Weather, Traffic, Solar, and Electricity datasets. LeRet consistently outperformed eight state-of-the-art models, including DLinear and LLM4TS.

Comprehensive performance comparison of LeRet against SOTA models. LeRet achieves the best (red) or second-best (blue) results in the vast majority of cases.

Key findings include:

  • SOTA Performance: LeRet ranked first in 89% of long-term forecasting experiments.
  • Efficiency: With only 0.15 million trainable parameters, LeRet is significantly more efficient than models like LLM4TS (7B parameters).
  • Generalization: In zero-shot tests, LeRet outperformed competitors by over 10%, demonstrating the robustness of its language-empowered features.

Conclusion

LeRet demonstrates that integrating linguistic priors with causal modeling can significantly enhance time series forecasting performance and efficiency. By keeping the LLM frozen and focusing on lightweight integration, LeRet provides a scalable and effective solution for complex forecasting tasks, particularly in data-constrained environments.

Summary updated

BibTeX
@inproceedings{huang2024leret,
  author = {Qihe Huang and Zhengyang Zhou and Kuo Yang},
  title = {{LeRet: Language-Empowered Retentive Network for Time Series Forecasting}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=I0XCsUB5Wi},
}
Figure