How Do Large Language Models Perform in Dynamical System Modeling

Xiao Luo, Binqi Chen, Haixin Wang, Zhiping Xiao, Ming Zhang, Yizhou Sun

Introduction

Dynamical system modeling—predicting the future states of interacting objects—is fundamental to fields ranging from physics to epidemiology. Traditionally, Graph Neural Networks (GNNs) have dominated this space due to their ability to capture spatial dependencies. However, GNNs often require extensive domain-specific training and struggle with generalization. This paper introduces LLM4DS, a benchmark designed to evaluate whether Large Language Models (LLMs), pre-trained on text, can reason about physical trajectories and multi-agent interactions in a zero-shot or few-shot manner.

Methodology: Bridging Numbers and Text

To enable LLMs to process numerical physical data, the researchers developed a specialized pipeline that translates trajectory matrices into text-based prompts.

The LLM4DS pipeline: converting numerical trajectories into text prompts for iterative state prediction.

The process involves:

  1. Prompt Engineering: Numerical observations are converted into strings. Crucially, "edge information" (which objects are interacting) is explicitly included in the prompt.
  2. Iterative Rollout: The model predicts the state at time $T+1$. This prediction is then appended to the history to predict $T+2$, allowing for long-term forecasting through successive steps.
  3. In-Context Learning: A "one-shot" strategy provides the model with a single example of a trajectory and its correct prediction to establish the output format.

The LLM4DS Benchmark

The authors curated nine datasets across three distinct domains to test the limits of LLM reasoning:

  • Physical Simulations: Springs (Hooke’s Law) and Charged particles.
  • Pedestrian Trajectories: The ETH-UCY benchmark, focusing on human movement in various environments.
  • Social Dynamics: Modeling the migration of opinions within a network.

Key Insights: Complexity and Relational Reasoning

The study revealed several critical characteristics of LLMs in physical modeling:

The Complexity Limit

While LLMs like Llama3-70B excel at modeling small systems (e.g., 4 interacting objects), their performance degrades sharply as the number of agents increases.

Mean Absolute Error (MAE) vs. Number of Nodes: LLMs show a significant performance drop as system complexity increases compared to specialized models like EGNN.

As shown in the figure above, while LLMs are competitive at low node counts, they struggle to parse the increasingly complex relational graphs of larger systems, leading to a spike in error.

The Necessity of Relational Context

A key finding was that LLMs do not merely perform numerical extrapolation. When "edge information" was removed from the prompts, performance plummeted. This indicates that the models actively use the provided relational context to inform their physical "reasoning." Furthermore, LLMs demonstrated strong zero-shot generalization, outperforming specialized models when tested on domains different from their training data.

Results and Limitations

On the Springs dataset, Llama3-70B achieved a 71.2% improvement in Mean Absolute Error (MAE) over state-of-the-art non-LLM models for short-term forecasts (under 7 steps). Additionally, with supervised fine-tuning, GPT-3.5 reached a precision of 0.973 in identifying underlying interactions between objects.

However, "error accumulation" remains a significant hurdle. Because the iterative rollout relies on previous predictions, small initial errors compound over time, leading to "hallucinated" trajectories in long-term scenarios. Future work will likely focus on integrating physical constraints, such as conservation of energy, directly into the decoding process to stabilize long-term predictions.

Conclusion

LLMs show remarkable promise as "zero-shot" physical simulators, particularly for short-term forecasting in low-complexity systems. While they currently face scaling limits and stability issues over long horizons, their ability to generalize across domains suggests a new path for flexible dynamical system modeling.

Summary updated

Abstract

This paper studies the problem of dynamical system modeling, which involves the evolution of multiple interacting objects. Recent data-driven methods often utilize graph neural networks (GNNs) to learn these interactions by optimizing the neural network in an end-to-end fashion. While large language models (LLMs) have shown exceptional zero-shot performance across various applications, their potential for modeling dynamical systems has not been extensively explored. In this work, we design prompting techniques for dynamical system modeling and systematically evaluate the capabilities of LLMs on two tasks, including dynamic forecasting and relational reasoning. An extensive benchmark LLM4DS across nine datasets is built for performance comparison. Our extensive experiments yield several key findings: (1) LLMs demonstrate competitive performance without training compared to state-of-the-art methods in dynamical system modeling. (2) LLMs effectively infer complex interactions among objects to capture system evolution. (3) Prompt engineering plays a crucial role in enabling LLMs to accurately understand and predict the evolution of systems.
BibTeX
@inproceedings{luo2025how,
  author = {Xiao Luo and Binqi Chen and Haixin Wang and Zhiping Xiao and Ming Zhang and Yizhou Sun},
  title = {{How Do Large Language Models Perform in Dynamical System Modeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.50},
}
Figure