Marrying LLMs with Dynamic Forecasting: A Graph Mixture-of-expert Perspective

Dapeng Jiang, Xiao LuoNAACL 2025 · DOI

Abstract

Predicting the evolution of dynamical systems is a fundamental challenge in science and engineering. While Graph Neural Networks (GNNs) have shown promise in modeling complex interactions, they often struggle with distribution shifts caused by changes in underlying environmental parameters. This paper introduces LEGO (LLM Judge with Graph Mixture-of-expert), a novel framework that integrates Large Language Models (LLMs) with Equivariant GNNs. By employing an LLM as a high-level "judge" to route inputs to specialized experts based on hierarchical text prompts, LEGO effectively bridges the gap between numerical precision and physical reasoning. Our evaluation across four diverse datasets demonstrates that LEGO significantly outperforms existing baselines, particularly in out-of-distribution scenarios.

Introduction

The ability to forecast the trajectories of dynamical systems—ranging from molecular dynamics to planetary motion—is critical for numerous applications. Graph Neural Networks (GNNs) are naturally suited for these tasks due to their ability to model relational data. However, standard GNNs are often "brittle"; they lack the capacity to generalize when the physical environment shifts (e.g., varying gravitational constants or spring stiffness). These shifts are often implicit in raw numerical data, making them difficult for purely data-driven models to identify.

Conversely, Large Language Models (LLMs) possess vast "world knowledge" and reasoning capabilities but struggle with the precise numerical generation required for 3D trajectory forecasting. This work proposes a hybrid approach: using LLMs not as predictors, but as context-aware routers that manage a mixture of specialized GNN experts.

Methodology: The LEGO Framework

The LEGO framework (LLM Judge with Equivariant Graph Mixture-of-expert) decomposes the forecasting task into a high-level reasoning component and a low-level numerical prediction component.

The LEGO architecture: Input states are processed by multiple experts, while an LLM Judge uses hierarchical prompts to perform context-aware routing for the final prediction.

Graph Mixture-of-Experts (MoE)

At the core of LEGO is a set of Equivariant GNN (EGNN) experts. Each expert is trained to specialize in different dynamical regimes. By maintaining a diverse pool of experts, the system can adapt to various environmental conditions.

Hierarchical Prompt Engineering

To enable the LLM to "reason" about the physical state, LEGO translates numerical environmental context into a hierarchical text format:

  1. System Level: Describes the overarching physical laws (e.g., "The dynamics follow Hooke's theory").
  2. Object Level: Details individual properties such as mass, initial velocity, and location.
  3. Edge (Link) Level: Defines the connectivity between objects (e.g., "Ball 1 is connected to Ball 2 and Ball 5").

LLM as a Judge

A pre-trained LLM (specifically Llama 3.1 8B) acts as the "Judge." It receives the hierarchical prompts along with candidate predictions from the GNN experts. Using Chain-of-Thought reasoning, the LLM evaluates which expert's prediction is most consistent with the described physical context and assigns routing weights accordingly. To maintain computational efficiency, the framework employs an alternative optimization strategy, updating routing weights and expert parameters in an alternating fashion rather than invoking the LLM at every time step.

Experimental Evaluation

LEGO was evaluated on four distinct datasets: Spring (N-body systems), Charged (Coulomb’s Law interactions), MD17 (molecular dynamics), and Motion (human motion capture). These datasets were chosen to test the model's ability to handle distribution shifts and transfer knowledge across different physical structures.

Key Results

The experimental results highlight LEGO's superior generalization capabilities:

  • Charged Dataset: LEGO achieved a 23.70% reduction in Mean Squared Error (MSE) compared to standard equivariant models, demonstrating its robustness to varying physical constants.
  • Molecular Dynamics (MD17): When transferring between different molecular structures (e.g., from salicylic acid to naphthalene), LEGO showed a 13.08% MSE reduction.
  • Zero-Shot Reasoning: The LLM Judge successfully identified the most appropriate experts even in zero-shot settings, providing logical justifications for its selections based on physical plausibility.
Visual comparison of the trajectory prediction generated by the LEGO framework (Ours).

Discussion and Conclusion

The success of LEGO stems from its "Reasoning over Regression" philosophy. By leveraging the LLM's ability to interpret metadata and context, the framework overcomes the limitations of purely numerical GNNs. The hierarchical prompting mechanism serves as a crucial bridge, allowing the LLM to utilize its pre-trained knowledge of physics to guide the selection of specialized numerical models.

While the computational overhead of LLM inference remains a consideration, the performance gains in out-of-distribution scenarios suggest that LEGO is a powerful step toward more robust and generalizable physical forecasting systems. Future work will explore scaling this approach to more complex, "black-box" simulations where physical laws are less explicitly defined.

Summary updated

BibTeX
@inproceedings{jiang2025marrying,
  author = {Dapeng Jiang and Xiao Luo},
  title = {{Marrying LLMs with Dynamic Forecasting: A Graph Mixture-of-expert Perspective}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.24},
}
Figure