Introduction
Dynamical system modeling—predicting the future states of interacting objects—is fundamental to fields ranging from physics to epidemiology. Traditionally, Graph Neural Networks (GNNs) have dominated this space due to their ability to capture spatial dependencies. However, GNNs often require extensive domain-specific training and struggle with generalization. This paper introduces LLM4DS, a benchmark designed to evaluate whether Large Language Models (LLMs), pre-trained on text, can reason about physical trajectories and multi-agent interactions in a zero-shot or few-shot manner.
Methodology: Bridging Numbers and Text
To enable LLMs to process numerical physical data, the researchers developed a specialized pipeline that translates trajectory matrices into text-based prompts.
The process involves:
- Prompt Engineering: Numerical observations are converted into strings. Crucially, "edge information" (which objects are interacting) is explicitly included in the prompt.
- Iterative Rollout: The model predicts the state at time $T+1$. This prediction is then appended to the history to predict $T+2$, allowing for long-term forecasting through successive steps.
- In-Context Learning: A "one-shot" strategy provides the model with a single example of a trajectory and its correct prediction to establish the output format.
The LLM4DS Benchmark
The authors curated nine datasets across three distinct domains to test the limits of LLM reasoning:
- Physical Simulations: Springs (Hooke’s Law) and Charged particles.
- Pedestrian Trajectories: The ETH-UCY benchmark, focusing on human movement in various environments.
- Social Dynamics: Modeling the migration of opinions within a network.
Key Insights: Complexity and Relational Reasoning
The study revealed several critical characteristics of LLMs in physical modeling:
The Complexity Limit
While LLMs like Llama3-70B excel at modeling small systems (e.g., 4 interacting objects), their performance degrades sharply as the number of agents increases.
As shown in the figure above, while LLMs are competitive at low node counts, they struggle to parse the increasingly complex relational graphs of larger systems, leading to a spike in error.
The Necessity of Relational Context
A key finding was that LLMs do not merely perform numerical extrapolation. When "edge information" was removed from the prompts, performance plummeted. This indicates that the models actively use the provided relational context to inform their physical "reasoning." Furthermore, LLMs demonstrated strong zero-shot generalization, outperforming specialized models when tested on domains different from their training data.
Results and Limitations
On the Springs dataset, Llama3-70B achieved a 71.2% improvement in Mean Absolute Error (MAE) over state-of-the-art non-LLM models for short-term forecasts (under 7 steps). Additionally, with supervised fine-tuning, GPT-3.5 reached a precision of 0.973 in identifying underlying interactions between objects.
However, "error accumulation" remains a significant hurdle. Because the iterative rollout relies on previous predictions, small initial errors compound over time, leading to "hallucinated" trajectories in long-term scenarios. Future work will likely focus on integrating physical constraints, such as conservation of energy, directly into the decoding process to stabilize long-term predictions.
Conclusion
LLMs show remarkable promise as "zero-shot" physical simulators, particularly for short-term forecasting in low-complexity systems. While they currently face scaling limits and stability issues over long horizons, their ability to generalize across domains suggests a new path for flexible dynamical system modeling.
Summary updated