VITRO: Vocabulary Inversion for Time-series Representation Optimization

Filippos Bellos, Nam H. Nguyen, Jason J. Corso

TL;DR

VITRO introduces "Vocabulary Inversion" to the world of time-series forecasting, allowing frozen Large Language Models (LLMs) to understand numerical data through a custom-built "pseudo-word" vocabulary. By learning dataset-specific embeddings instead of relying on standard text tokens, VITRO achieves state-of-the-art performance in long-term forecasting without needing to fine-tune the underlying model.

What Problem Does This Paper Address?

Large Language Models (LLMs) are surprisingly good at reasoning, but they have a fundamental "language barrier" when it comes to time-series data. LLMs are built on vocabularies designed for discrete, symbolic text (words and sub-words). Time-series data, however, is continuous and numerical.

Current state-of-the-art methods try to bridge this gap by "reprogramming" time-series patches into text-like prototypes. However, these methods are still tethered to the original LLM vocabulary, which was never meant to capture the nuanced temporal dynamics or seasonal patterns found in electricity loads or traffic flow. The authors ask: Is there a better way to represent time-series data than using the general-purpose vocabulary of LLMs?

Methods

The authors propose VITRO (Vocabulary Inversion for Time-series Representation Optimization), inspired by "textual inversion" techniques used in image generation (like Stable Diffusion). The process happens in two stages:

  1. Stage 1: Vocabulary Inversion: Instead of using existing words, VITRO learns a set of "pseudo-words"—new embedding vectors ($v_i$) that represent specific time-series instances and a shared "domain" embedding ($s$) that captures the essence of the entire dataset. These are optimized through an iterative process to minimize forecasting error while keeping the LLM itself frozen.
  2. Stage 2: Forecasting: Once the vocabulary is learned, it is used to guide the LLM. The authors test two ways to use this new dictionary:
  3. VITRO-Sim: A similarity-based approach that selects the most relevant "core lexicon" embeddings for each data patch.
  4. VITRO-TimeLLM: An attention-based approach that integrates the new vocabulary into the existing Time-LLM framework.

The pipeline involves normalizing the data (RevIN), breaking it into patches, and projecting those patches into the LLM's embedding space as "prompts" alongside the new pseudo-words.

Datasets

The researchers evaluated VITRO on seven widely recognized public benchmarks for long-term forecasting:

  • Weather: Regional climate data.
  • Electricity: Hourly consumption records.
  • Traffic: Road occupancy rates.
  • ETT (Electricity Transformer Temperature): Four separate datasets (ETTh1, ETTh2, ETTm1, ETTm2) reflecting different recording frequencies.

Key Insights and Ideas

  • The Manifold Discovery: Qualitative analysis (PCA and t-SNE) revealed that VITRO’s learned vocabulary forms a structured, U-shaped manifold. In contrast, standard LLM text embeddings look like a diffuse, circular cloud. This suggests VITRO is successfully capturing the underlying geometric structure of temporal data.
  • Horizontal Striping: Heatmaps of the attention weights showed distinct "horizontal striping." This indicates that specific pseudo-words consistently anchor themselves to certain parts of the input sequence, effectively acting as specialized temporal markers.
  • Frozen is Better: VITRO outperforms methods that require partial fine-tuning of the LLM (like $S^2$IP-LLM). This suggests that the "knowledge" inside the LLM is already sufficient; we just need a better way to prompt it.

Key Results

VITRO-enhanced methods consistently outperformed both their "standard vocabulary" counterparts and traditional time-series models:

  • Consistent Gains: Replacing the standard vocabulary with VITRO’s learned embeddings led to improvements across all 7 datasets for both MSE (Mean Squared Error) and MAE (Mean Absolute Error).
  • State-of-the-Art: VITRO achieved the best results in most categories, particularly on the ETT datasets. For example, on ETTh1, VITRO-Sim achieved an MSE of 0.412, outperforming the standard Time-LLM (0.437) and the popular PatchTST (0.444).
  • Versatility: The method proved effective across different LLM backbones, including GPT-2 and Llama-7B.

Limitations and Future Work

  • Optimization Cost: Because VITRO relies on an iterative optimization process to learn the vocabulary, the computational "entry fee" can be high for extremely large datasets compared to simple linear projections.
  • Task Expansion: Currently focused on long-term forecasting, the authors suggest that future work should explore whether these learned vocabularies can improve performance in classification, anomaly detection, and imputation.
  • Integration: There is potential to integrate VITRO with other emerging LLM-based time-series architectures to see if the benefits are additive.

Summary updated

BibTeX
@inproceedings{bellos2025vitro,
  author = {Filippos Bellos and Nam H. Nguyen and Jason J. Corso},
  title = {{VITRO: Vocabulary Inversion for Time-series Representation Optimization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10889449},
}
Figure