TL;DR
While Large Language Models (LLMs) are surprisingly good at time series forecasting, they often suffer from "near-sightedness," focusing on local patterns while missing global trends. FreqLLM solves this by injecting frequency-domain information into the model and aligning it with the LLM’s internal semantic space, resulting in state-of-the-art accuracy and exceptional performance in data-scarce (few-shot) scenarios.
The Challenge: Overcoming "Near-Sightedness" in LLMs
In the world of Time Series Forecasting (TSF), researchers have recently turned to LLMs like GPT-2 to act as powerful pattern recognizers. However, most existing approaches feed the LLM raw time-domain data. The authors discovered a major flaw in this: time-domain inputs lead to "diagonal correlations," where the model only pays attention to a few adjacent time steps. This makes it difficult for the model to capture global periodicity, long-term trends, and the "big picture" of the data.
Furthermore, there is a fundamental "modality gap." LLMs are designed to process discrete tokens (words), while time series data is continuous. Simply "patching" numbers into a model built for language often leads to an incomplete understanding of the underlying signal.
Methodology: The FreqLLM Framework
FreqLLM bridges the gap between signal processing and natural language processing through three main modules, while keeping the LLM backbone (GPT-2) frozen for computational efficiency.
- Dual-Scale Frequency-Domain Encoding: Instead of just looking at the raw sequence, FreqLLM extracts frequency signals at two scales. It uses a Global FFT for the entire time window and a "Local" frequency extraction (using a sliding window and target attention) to capture recent, high-frequency changes near the prediction point.
- Frequency-Domain Semantic Alignment: To help the LLM "understand" these signals, the authors map frequency embeddings to a set of Semantic Exemplars. These are essentially "soft prompts" derived from pre-trained word embeddings that have been filtered to be relevant to time series tasks.
- Dual-Domain Reprogramming: This module acts as a translator. It uses a multi-head cross-attention mechanism to "reprogram" both time-domain and frequency-domain signals into the semantic space of the LLM. This ensures the continuous signal is formatted in a way that the frozen LLM backbone can easily interpret.
Experimental Results
The researchers evaluated FreqLLM on a comprehensive suite of standard TSF benchmarks, including the ETT family, Weather, Traffic, and the M4 benchmark.
- Superior Accuracy: FreqLLM outperformed state-of-the-art task-specific models like PatchTST by 3.83% on average across long-term tasks.
- Beating other LLM-TSF models: It showed a 2.21% improvement over Time-LLM and a 3.88% improvement over GPT4TS.
- Few-Shot Dominance: In scenarios with very limited data (5% training set), FreqLLM significantly outperformed traditional models. This suggests that the frequency-domain alignment helps the model "understand" the nature of the data much faster than learning from raw sequences alone.
- Short-Term Success: On the M4 dataset, it achieved a 3.15% improvement over Time-LLM, proving it isn't just a long-term specialist.
Key Insights
- Visualizing the "Aha!" Moment: Visualizing the model's attention weights confirms that adding frequency signals transforms narrow diagonal patterns into "curvilinear" ones, showing that the model is finally "seeing" connections across much longer time horizons.
- Frequency as a Denoiser: By separating low-frequency (global trends) from high-frequency (local variations) components, the model can ignore redundant noise that often trips up purely time-domain models.
- Language as a Scaffold: By using word embeddings as "exemplars," the model leverages the rich, pre-trained relational knowledge of the LLM to organize and interpret numerical data.
Conclusion and Future Work
FreqLLM demonstrates that frequency awareness is a critical component for adapting LLMs to time series tasks. While the current framework is a significant step forward, future work could involve "multimodal" pre-training where the model is exposed to both text and time series data simultaneously. Additionally, exploring more streamlined ways to integrate frequency data could improve inference speed, while applying this approach to other sequential data modalities remains a promising avenue for research.
Summary updated