Abstract
Large Language Models (LLMs) are increasingly deployed as general-purpose estimators, yet their ability to internalize the underlying physical rules of a domain—"world models"—remains poorly understood. This paper introduces LEVERWORLDS, a synthetic framework designed to evaluate how efficiently LLMs learn physical structure versus specific data distributions. Our findings reveal a significant "efficiency gap": while Transformers can eventually learn physical rules, they require orders of magnitude more data than classical statistical models with appropriate inductive biases.
Introduction
In statistical learning, a fundamental tension exists between discovering persistent structures (universal rules) and modeling context-dependent variability (stochastic details). While LLMs like the OPT family or GPT-4o are highly flexible, the bias-variance tradeoff suggests that this lack of prior assumptions may lead to high variance and poor sample efficiency in structured environments. This study investigates whether the massive flexibility of LLMs makes them "lazy" learners when faced with simple, deterministic physical worlds compared to classical algorithms like Logistic Regression.
The LEVERWORLDS Framework
The researchers developed LEVERWORLDS, a synthetic environment centered on a physical lever system. The outcome—whether the lever tips Left (L) or Right (R)—is governed by the torque formula: $T = \text{side} \cdot \text{distance} \cdot \text{mass}$.
The framework generates text-based descriptions of physical setups (e.g., "object1 mass: 3, distance: 2... balance: L"). It includes variations where certain variables, such as density or volume, are latent, requiring the model to infer hidden distributions alongside the physical laws.
Experimental Methodology
The study compared several learning paradigms:
- Baseline Statistical Methods: Naive Maximum Likelihood Estimation (MLE) and Logistic Regression (LR) with polynomial features.
- Transformer Fine-tuning: OPT models (125M to 6.7B parameters) were fine-tuned using LoRA on text-represented physical observations.
- In-Context Learning (ICL): Prompting frontier models like GPT-4o with examples to predict probabilities for new instances.
- Pipeline Learning: A hybrid approach where the LLM generates a Python parsing function to feed data into a classical solver.
Key Results and Insights
The experiments highlighted a stark contrast in sample efficiency. In a simple setup ("World-1"), Logistic Regression achieved low error with approximately 100 samples. In contrast, OPT models required over 2,500 samples to reach comparable performance.
The Efficiency Gap
As shown in the figure above, while larger models (6.7B) generally exhibit better sample efficiency initially, the TV distance for all Transformer models remains significantly higher than classical baselines for small sample sizes. This suggests that LLMs must spend thousands of samples ruling out incorrect hypotheses that a structured model ignores by design.
Structure vs. Variability
The authors found that LLMs struggle more with learning the underlying structure (the physics) than the variability (the distribution of weights). Once the physical rules are internalized, the task becomes significantly easier. However, the "general-purpose" nature of Transformers acts as a double-edged sword; without an inherent physical prior, they are "sample hungry."
LLMs as Orchestrators
The "Pipeline Learning" experiment showed that GPT-4o is more effective at setting up a classical solver (writing code to parse and solve) than it is at being the solver itself. This suggests that the future of world modeling may lie in hybrid architectures that combine the linguistic flexibility of LLMs with the strong inductive biases of symbolic or statistical methods.
Conclusion
LEVERWORLDS demonstrates that while LLMs are capable of learning world models, they are fundamentally less efficient than models with domain-specific inductive biases. Future research should explore architectures that can more seamlessly integrate these biases, moving beyond "bigger is better" toward more efficient, structured reasoning.
Summary updated