LLMODE: Aligning ODEs with LLMs via Gated Token Injection for Irregular Spatio-Temporal Forecasting

Di Zhang, Jingyang Zhang, Ziqian Wang, Chi Zhang, Yikun Ban, Ziwei Zhang, Ruijie WangarXiv 2026 · arXiv

TL;DR

LLMODE is a novel forecasting framework that connects Neural Ordinary Differential Equations (ODEs) with frozen Large Language Models (LLMs) to handle irregular, asynchronous spatio-temporal data. By compressing continuous latent trajectories into a fixed budget of memory tokens and injecting them via gated cross-attention, it achieves state-of-the-art accuracy while reducing token usage by up to 89.2% and boosting inference speed by 7.7× compared to existing LLM forecasters.

What Problem Does This Paper Address?

Spatio-temporal forecasting—predicting systems that evolve over both space and time, like traffic flows, crime patterns, or particle dynamics—is notoriously difficult when data arrives irregularly. In the real world, sensors fail, events occur at non-uniform timestamps, and updates across different locations are rarely synchronized.

While LLMs have recently been adapted for spatio-temporal forecasting by encoding data streams into "soft tokens," existing models struggle with irregular data for three main reasons:

  1. Temporal Asynchrony: Standard tokenization assumes a regular time grid, which obscures actual time gaps and missing intervals.
  2. Context-Length Constraints: Exposing long, fine-grained history to an LLM triggers the quadratic computational cost of self-attention, bloating the context window with redundant data.
  3. Representation Alignment: Raw numerical trajectories do not natively align with the semantic space an LLM learns during language pretraining.

The authors introduce LLMODE to bridge the gap between continuous-time physical modeling (via Neural ODEs) and high-capacity sequence prediction (via frozen LLMs) without melting your GPU budget.

Methods

Figure 2: Framework of LLMODE. LLMODE constructs two memory streams to condition a frozen LLM backbone. The graph-aware ODE encoder first reconstructs a continuous-time latent trajectory, which is resampled into a fixed number of dynamic memory tokens. In parallel, compact statistical descriptors are encoded and resampled into context memory tokens. Prompt hidden states retrieve information from both memories through dual-source gated cross-attention, and the final query-slot states are mapped to forecasts.

As illustrated in the framework above, LLMODE processes irregular data through a decoupled, three-stage pipeline:

  1. Graph-Aware ODE Encoder: Instead of forcing data onto a rigid time grid, a Neural ODE evolves latent states continuously based on actual elapsed time. When a new observation arrives, a Gated Recurrent Unit (GRU) corrects the state, which is then refined using a graph neural network to capture spatial dependencies among neighboring nodes.
  2. Fixed-Budget Perceiver Resampler: To prevent context bloat, the continuous trajectory is distilled into a small, fixed number ($K$) of dynamic memory tokens. A dynamics-aware attention bias forces the resampler to prioritize intervals with rapid, highly informative changes. Concurrently, global statistical summaries are serialized as text and resampled into context memory tokens.
  3. Dual-Source Gated Cross-Attention: Instead of directly concatenating these tokens to the text prompt (which can dilute information), LLMODE injects them into selected layers of a frozen LLM (Vicuna-7B) using learnable gates initialized to zero. This ensures the LLM's pretrained representations are modified in a controlled, prediction-aware manner.

Datasets

The framework was validated across five distinct benchmarks:

  • Urban Datasets (NYC): NYCtaxi, NYCbike, and NYCcrime. These represent event-level, irregularly sampled real-world data aggregated into regional graph structures.
  • Physics Simulations: Springs and Charged. These benchmarks simulate 2D interacting particle systems under asynchronous, partial observations to test the model's grasp of complex physical dynamics.

Key Insights and Ideas

  • Continuous Beats Discrete Under Sparsity: As data becomes scarcer, discrete encoders degrade rapidly. Neural ODEs excel here because they model the underlying system dynamics continuously across long, unobserved intervals.
  • Information is Concentrated: Exposing the LLM to the full trajectory adds massive computational overhead without improving accuracy. A tiny budget of adaptively selected tokens ($K=5$ or $8$) captures virtually all forecasting-relevant information.
  • The Power of Frozen Representations: Replacing the frozen LLM with an MLP or a Transformer trained from scratch maintains supervised performance but tanks zero-shot accuracy on unseen regions. The pretrained language backbone provides a robust structural prior that is essential for spatial generalization.

Key Results

Figure 3: Forecasting performance under different missing rates on the Springs system.
  • Superior Accuracy: LLMODE outperformed 11 competitive baselines (including UrbanGPT and GPT4TS) across all five datasets, showing a widening advantage as data missing rates increased up to 80% (see the Springs system plot above).
  • Massive Efficiency Gains: On the NYCbike zero-shot task, LLMODE reduced token consumption by 89.2% and delivered a 7.7× inference speedup over UrbanGPT.
  • Strong Zero-Shot Generalization: When evaluated on entirely unseen urban regions without any fine-tuning, LLMODE achieved an average relative improvement of 11.4% over the best baseline, proving it learns transferable dynamics rather than region-specific memorization.

Limitations and Future Work

While LLMODE drastically reduces the computational footprint of LLM-based forecasting, a notable efficiency gap remains when compared to lightweight, non-LLM baselines (like MTGNN or tPatchGNN), which boast significantly lower inference latencies. The authors note that future work should focus on further optimizing the computational efficiency of the LLM backbone itself to make these high-capacity forecasters more viable for real-time deployment.

Summary updated

Abstract

Large language models (LLMs) have shown promise for spatio-temporal forecasting, but existing approaches often rely on regularly sampled token sequences and struggle with irregular observations because of temporal asynchrony, representation-space misalignment, and limited context windows. We propose LLMODE, a token-efficient framework for irregular spatio-temporal forecasting with a frozen LLM backbone. LLMODE first uses a graph-aware ODE encoder to reconstruct irregular graph observations as a continuous-time latent trajectory. A Fixed-Budget Perceiver Resampler then compresses this variable-length trajectory into a fixed number of dynamic memory tokens. In parallel, compact statistical descriptors are encoded and resampled into context memory tokens. A dual-source gated cross-attention module injects both memories into the frozen LLM, enabling controlled utilization of external spatio-temporal evidence. Experiments on three real-world urban datasets and two physical-dynamics benchmarks show competitive overall performance, with clearer advantages under sparse or dynamically complex irregular sampling. Additional evaluations on unseen urban regions further demonstrate strong zero-shot generalization without adaptation.
BibTeX
@misc{zhang2026llmode,
  author = {Di Zhang and Jingyang Zhang and Ziqian Wang and Chi Zhang and Yikun Ban and Ziwei Zhang and Ruijie Wang},
  title = {{LLMODE: Aligning ODEs with LLMs via Gated Token Injection for Irregular Spatio-Temporal Forecasting}},
  year = {2026},
  eprint = {2608.29640},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.29640},
}
Figure