Abstract
Although the rise of large language models (LLMs) has introduced new opportunities for time series forecasting, existing LLM-based solutions require excessive training and exhibit limited transferability. In view of these challenges, we propose TimeRAG, a framework that incorporates Retrieval-Augmented Generation (RAG) into time series forecasting LLMs, which constructs a time series knowledge base from historical sequences, retrieves reference sequences from the knowledge base that exhibit similar patterns to the query sequence measured by Dynamic Time Warping (DTW), and combines these reference sequences and the prediction query as a textual prompt to the time series forecasting LLM. Experiments on datasets from various domains show that the integration of RAG improved the prediction accuracy of the original model by 2.97% on average.
Introduction
Large Language Models (LLMs) have revolutionized natural language processing, but their application to time series forecasting remains challenging. Current LLM-based forecasting solutions often require expensive fine-tuning and struggle to transfer across different domains (e.g., from financial data to weather data). Furthermore, LLMs are prone to "hallucinations"—generating outliers or fabricated patterns that do not align with actual historical trends. Because standard LLMs lack a mechanism to reference similar historical examples during inference, they rely entirely on internal weights, which may not capture the specific nuances of a new dataset.
Methodology
To address these limitations, we introduce TimeRAG, a framework that enhances LLMs by providing them with relevant historical context. The process consists of three main stages:
- Knowledge Base Construction: Instead of storing every raw data point, historical sequences are sliced into segments using a sliding window. K-means clustering is then applied to these segments, and only the cluster centroids are stored. This creates a compact, representative knowledge base of common waveforms and trends.
- Pattern Retrieval: When a new query sequence arrives, the system searches the knowledge base for the most similar patterns using Dynamic Time Warping (DTW). Unlike standard Euclidean distance, DTW is "shape-aware" and tolerant of temporal distortions, such as patterns that are slightly stretched or shifted in time.
- Forecasting via Prompting: The retrieved "referential sequences" are combined with the input query into a natural language prompt. This prompt is fed into an LLM (utilizing a Llama3-based model with a reprogramming layer) to align numerical data with text tokens. The LLM uses these examples as a guide to predict the next steps.
Experimental Setup
The framework was evaluated on the M4 dataset, a massive benchmark containing 100,000 time series from various domains across six different frequencies: Yearly, Quarterly, Monthly, Weekly, Daily, and Hourly. This variety allowed for testing how well TimeRAG adapts to different temporal granularities and domain-specific patterns.
Key Results and Discussion
TimeRAG demonstrates that providing LLMs with "what similar situations looked like in the past" significantly improves forecasting performance.
- Accuracy Boost: TimeRAG improved the prediction accuracy of the baseline model (Time-LLM) by 2.97% on average.
- Domain Success: The most significant gains were seen in "Weekly" frequency data, where the model achieved a 13.12% improvement in Mean Absolute Scaled Error (MASE).
- State-of-the-Art Performance: TimeRAG outperformed several specialized models, including iTransformer, FEDformer, and DLinear. In 18 different comparative analyses across various metrics and frequencies, TimeRAG secured a "top-three" position 14 times.
Conclusion
TimeRAG offers a "zero-parameter enhancement" for existing models like Time-LLM, improving performance without modifying foundational parameters. While the retrieval step adds some computational overhead, the use of K-means clustering makes the process feasible for real-time applications. Future work will explore more advanced prompt engineering strategies and enhanced sequential data management to handle even larger, more complex knowledge bases.
Summary updated