Evaluating LLM-Generated World Models for Neuro-Symbolic Reasoning

Souhaila Djaffal, Amna Jamal, Gagana Madhugiri Devaraja, Anidipta Pal, Jobst HeitzigarXiv 2026 · arXiv

TL;DR

LLMs are surprisingly adept at generating "mechanically valid" formal world models that execute without errors, but they frequently fail to capture the actual logic of the scenarios they are meant to represent. This paper demonstrates that executability is a poor proxy for faithfulness, as valid models often hallucinate extra complexity or omit critical stakeholders, potentially leading to silent failures in safety-critical neuro-symbolic systems.

What Problem Does This Paper Address?

Neuro-symbolic AI aims to combine the learning power of neural networks with the rigorous reasoning of symbolic logic. For these systems to perform safety-critical tasks—like verifying if an action is reversible or if a human remains "empowered"—they require a formal world model (a mathematical representation of states, actions, and consequences).

Manually coding these models for every new situation is a bottleneck. While Large Language Models (LLMs) can theoretically translate natural language descriptions into formal code (like Python or JSON), we don't yet know if we can trust their output. The core challenge is that a generated model might be syntactically perfect and "run" correctly, yet represent the wrong decision problem. If the world model is unfaithful, even a perfect downstream planner will make unsafe or incorrect decisions.

Methods

The authors propose a three-lane evaluation framework to move beyond simple code-correctness:

  1. Lane 1: Mechanical Validity: Does the code run? Are transition probabilities normalized to 1? Are there "dead states"?
  2. Lane 2: Structural Fidelity: Does the generated state-transition graph match a ground-truth reference? The authors use a permutation-invariant F1 score to compare models even if the LLM uses different names for states or actions.
  3. Lane 3: Semantic Diagnostics: Does the model include "must-have" features (e.g., a specific stakeholder) and avoid "must-not-have" assumptions?

They prompted five state-of-the-art LLMs (including GPT-5.6 Sol and Claude Opus 5) to generate Partially Observable Stochastic Games (POSGs)—a complex formalization that accounts for multiple agents and hidden information.

Datasets

The researchers introduced the world-model-suite, a benchmark of 44 scenarios across four tiers:

  • Toy Environments: 10 cases testing basic mechanics (preconditions, terminal states).
  • Simple Games: 9 cases with known rules and authored reference structures.
  • Situational Scenarios: 11 open-ended cases involving safety trade-offs and scarce resources.
  • Cultural Scenarios: 14 cases where social norms might be underspecified, testing for cultural bias.

Key Insights and Ideas

  • The "Executability Trap": Mechanical validity is high across the board, but structural agreement is dismal. In the game track, nearly all models passed the mechanical audit, but the mean structural F1 score was only 0.366.
  • Hallucinated Complexity: LLMs often "over-engineer" models. For example, Claude Opus 5 frequently added spurious states and actions not found in the prompt (see Figure 2).
  • Bias at the Level of Consequence: In cultural scenarios, simply naming a country (e.g., "Riyadh, Saudi Arabia") caused LLMs to fundamentally restructure the world model—changing the number of terminal states and available actions—even when automated semantic checks showed no "toxic" language. This suggests that bias in world models can be structural rather than just lexical.
Figure 2: Compressed phase graph for G6 versus the mechanically valid Claude Opus 5 artifact from run 1 (matched F1 0.0383). The candidate has 101 states. Unlike the two-bid-level reference, it introduces bids {0, 1, 2, 3} despite the rubric explicitly disallowing invented bid amounts, and expands the dynamics through four type states and 16 tie states. Nodes are grouped by verified phase prefix for legibility.
  • Case Identity Trumps Model Identity: The specific difficulty of a scenario was a stronger predictor of failure than which specific LLM was used. If one model struggled with a complex game, they usually all did.

Key Results

The quantitative gap between "running" and "being right" is stark:

Track Lane 1: Mechanical (Pass Rate) Lane 2a: Structural (Mean F1)
Game 134/135 0.3660
Situational 104/135 0.4324
Cultural 125/140 0.5533

Data from Table 1 (Figure 5).

Only 28 out of 134 mechanically valid game models exactly matched the authored reference. Furthermore, as shown in the comparison of LLMs (Table 2/Figure 6), models like GPT-5.6 Sol and Llama 3.3 70B performed significantly better at capturing structure than Claude Opus 5, despite all models producing runnable code.

Table 2: Permutation-invariant structural comparison over 27 final artifacts per LLM. Means are supplemented by median and IQR; machine-readable results also report game-clustered bootstrap intervals.

Limitations and Future Work

  • Depth of Comparison: Structural comparisons were limited to a depth of three transitions. Divergences that occur later in a game might be missed.
  • Human-in-the-Loop: The "Semantic Review" (Lane 3) currently relies on programmatic checks; the authors note that full human validation of these features is an ongoing next step.
  • Downstream Impact: The paper evaluates the models, but not yet the policies generated from them. Future work will investigate how specific structural errors (like a missing terminal state) translate into actual unsafe agent behavior.
  • Scale: The benchmark is hand-crafted and relatively small, intended for deep qualitative analysis rather than massive statistical scaling.

Summary updated

Abstract

Formal world models can support planning, verification, and safety reasoning, but constructing them from open-ended natural-language situations is difficult, particularly in multi-agent settings where several formalizations may be plausible. We study whether five LLMs can generate such world models as partially observable stochastic games and whether mechanical validity is sufficient to establish that the resulting model faithfully represents the intended decision problem. We introduce a benchmark of 44 scenarios spanning game, situational, and cultural settings and evaluate generated world models along three dimensions: mechanical validity, reference-relative structural agreement, and semantic criteria. Across three empirical tracks, we analyze 410 first-pass generations. In the reference-backed game track, 134/135 generated world models pass a strict mechanical audit, yet mean permutation-invariant structural F1 is only 0.366, with just 28/134 mechanically valid models exactly matching the authored reference. In the cultural track, explicitly naming the country changes the generated structural representation for most LLMs while producing only small changes in automated semantic scores. These results show that executability is a necessary but insufficient criterion for LLM-generated world models: models that are mechanically valid can still differ substantially in the structure used by downstream neuro-symbolic reasoning and planning.
BibTeX
@misc{djaffal2026evaluating,
  author = {Souhaila Djaffal and Amna Jamal and Gagana Madhugiri Devaraja and Anidipta Pal and Jobst Heitzig},
  title = {{Evaluating LLM-Generated World Models for Neuro-Symbolic Reasoning}},
  year = {2026},
  eprint = {2611.14419},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14419},
}
Figure