Abstract
Time series forecasting (TSF) plays a pivotal role in many real-world applications. Recently, the utilization of Large Language Models (LLM) in TSF has demonstrated exceptional predictive performance, surpassing most task-specific forecasting models. The success of LLM-based forecasting methods underscores the importance of causal dependence modeling and pre-trained knowledge transfer. However, challenges persist in directly applying LLM to TSF, i.e., the unacceptable parameter scales for resource-intensive model optimization, and the significant gap of feature space between structural numerical time series and natural language. To this end, we propose LeRet, a Language-empowered Retentive network for TSF. Technically, inspired by the causal extraction in LLM, we propose a causal dependence learner, enhanced by a patch-level pre-training task, to capture sequential causal evolution. To minimize the gap between numeric and language, we initialize a language description protocol for time series and design a TS-related language knowledge extractor to learn from language description, avoiding training with large-scale parameters. Finally, we dedicatedly achieve a Language-TS Modality Integrator for the fusion of two types data, and enable language-empowered sequence forecasting. Extensive evaluations demonstrate the effectiveness of our LeRet, especially reveal superiority on few-shot, and zero-shot forecasting tasks.
Introduction
Time series forecasting (TSF) is a cornerstone of modern decision-making, essential for domains ranging from energy management to traffic optimization. While traditional task-specific models have been the standard, recent research has pivoted toward Large Language Models (LLMs). These models leverage vast pre-trained "world knowledge" to achieve remarkable generalization, particularly in zero-shot scenarios.
However, the direct application of LLMs to TSF faces two critical bottlenecks: the Parameter Wall and the Modality Gap. Billions of parameters make LLMs computationally prohibitive for many practical applications, and the discrete nature of text tokens is fundamentally at odds with continuous numerical time series. LeRet (Language-Empowered Retentive Network) addresses these issues by combining the causal modeling efficiency of Retentive Networks with a lightweight language knowledge extractor, achieving state-of-the-art performance with a fraction of the parameters (0.15M vs. 7B).
The LeRet Framework
LeRet introduces a novel architecture designed to respect the causal nature of time while bridging the gap between numerical and semantic data. The framework consists of four primary modules:
- Causal Dependence Learner (CDL): Moving away from standard bidirectional attention, LeRet utilizes a Retentive Network (RetNet) backbone. This mechanism employs a decay factor to prioritize recent history, strictly adhering to sequential causality—a more natural fit for time-ordered data.
- TS-Related Language Knowledge Extractor (TRLE): To capture semantic priors without the overhead of a full LLM, LeRet uses a "Language Description Protocol." High-level features like trend, periodicity, and noise are described in natural language (e.g., "This time series shows an overall upward trend"). A frozen LLM (LLaMA) then extracts embeddings from these descriptions.
- Language-TS Modality Integrator (LTMI): A two-stage cross-attention module maps text embeddings into the time series feature space and fuses them with numerical features, empowering the raw data with semantic context.
- Sequence Forecasting with Patch-level Enhancement: The model is pre-trained on an autoregressive task—predicting the next "patch" of data—to help the causal learner understand growth patterns before the final forecasting task.
Experimental Evaluation
LeRet was evaluated across 64 benchmarks, including standard long-term forecasting (ETT, Weather, Traffic, Solar, Electricity) and short-term forecasting (PEMS collection). The model's generalization capabilities were further tested in zero-shot and few-shot (10% training data) settings.
Key Results
- Dominant Performance: LeRet achieved top-1 performance in 57 out of 64 settings. As shown in the results table, it consistently outperforms both task-specific models like ModernTCN and massive LLM-based models like LLM4TS and GPT4TS.
- Superior Generalization: In zero-shot scenarios, LeRet outperformed state-of-the-art models by over 10%. In few-shot learning, it maintained a significant lead over much larger competitors.
- Extreme Efficiency: Despite its performance, LeRet requires only 0.15 million parameters during training and inference, making it several thousand times smaller than models like LLM4TS (7B parameters).
Conclusion and Future Work
LeRet demonstrates that respecting the "arrow of time" through causal retention and incorporating semantic priors can yield superior forecasting results without the need for massive parameter counts. While the current manual selection of language descriptions is a limitation, future work will explore automated, dynamic description generation. Additionally, refining the modality fusion between text and numbers remains a promising avenue for further enhancing forecasting accuracy across diverse datasets.
Summary updated