TL;DR
LLMs are surprisingly adept at generating "mechanically valid" formal world models that execute without errors, but they frequently fail to capture the actual logic of the scenarios they are meant to represent. This paper demonstrates that executability is a poor proxy for faithfulness, as valid models often hallucinate extra complexity or omit critical stakeholders, potentially leading to silent failures in safety-critical neuro-symbolic systems.
What Problem Does This Paper Address?
Neuro-symbolic AI aims to combine the learning power of neural networks with the rigorous reasoning of symbolic logic. For these systems to perform safety-critical tasks—like verifying if an action is reversible or if a human remains "empowered"—they require a formal world model (a mathematical representation of states, actions, and consequences).
Manually coding these models for every new situation is a bottleneck. While Large Language Models (LLMs) can theoretically translate natural language descriptions into formal code (like Python or JSON), we don't yet know if we can trust their output. The core challenge is that a generated model might be syntactically perfect and "run" correctly, yet represent the wrong decision problem. If the world model is unfaithful, even a perfect downstream planner will make unsafe or incorrect decisions.
Methods
The authors propose a three-lane evaluation framework to move beyond simple code-correctness:
- Lane 1: Mechanical Validity: Does the code run? Are transition probabilities normalized to 1? Are there "dead states"?
- Lane 2: Structural Fidelity: Does the generated state-transition graph match a ground-truth reference? The authors use a permutation-invariant F1 score to compare models even if the LLM uses different names for states or actions.
- Lane 3: Semantic Diagnostics: Does the model include "must-have" features (e.g., a specific stakeholder) and avoid "must-not-have" assumptions?
They prompted five state-of-the-art LLMs (including GPT-5.6 Sol and Claude Opus 5) to generate Partially Observable Stochastic Games (POSGs)—a complex formalization that accounts for multiple agents and hidden information.
Datasets
The researchers introduced the world-model-suite, a benchmark of 44 scenarios across four tiers:
- Toy Environments: 10 cases testing basic mechanics (preconditions, terminal states).
- Simple Games: 9 cases with known rules and authored reference structures.
- Situational Scenarios: 11 open-ended cases involving safety trade-offs and scarce resources.
- Cultural Scenarios: 14 cases where social norms might be underspecified, testing for cultural bias.
Key Insights and Ideas
- The "Executability Trap": Mechanical validity is high across the board, but structural agreement is dismal. In the game track, nearly all models passed the mechanical audit, but the mean structural F1 score was only 0.366.
- Hallucinated Complexity: LLMs often "over-engineer" models. For example, Claude Opus 5 frequently added spurious states and actions not found in the prompt (see Figure 2).
- Bias at the Level of Consequence: In cultural scenarios, simply naming a country (e.g., "Riyadh, Saudi Arabia") caused LLMs to fundamentally restructure the world model—changing the number of terminal states and available actions—even when automated semantic checks showed no "toxic" language. This suggests that bias in world models can be structural rather than just lexical.
- Case Identity Trumps Model Identity: The specific difficulty of a scenario was a stronger predictor of failure than which specific LLM was used. If one model struggled with a complex game, they usually all did.
Key Results
The quantitative gap between "running" and "being right" is stark:
| Track | Lane 1: Mechanical (Pass Rate) | Lane 2a: Structural (Mean F1) |
|---|---|---|
| Game | 134/135 | 0.3660 |
| Situational | 104/135 | 0.4324 |
| Cultural | 125/140 | 0.5533 |
Data from Table 1 (Figure 5).
Only 28 out of 134 mechanically valid game models exactly matched the authored reference. Furthermore, as shown in the comparison of LLMs (Table 2/Figure 6), models like GPT-5.6 Sol and Llama 3.3 70B performed significantly better at capturing structure than Claude Opus 5, despite all models producing runnable code.
Limitations and Future Work
- Depth of Comparison: Structural comparisons were limited to a depth of three transitions. Divergences that occur later in a game might be missed.
- Human-in-the-Loop: The "Semantic Review" (Lane 3) currently relies on programmatic checks; the authors note that full human validation of these features is an ongoing next step.
- Downstream Impact: The paper evaluates the models, but not yet the policies generated from them. Future work will investigate how specific structural errors (like a missing terminal state) translate into actual unsafe agent behavior.
- Scale: The benchmark is hand-crafted and relatively small, intended for deep qualitative analysis rather than massive statistical scaling.
Summary updated