Abstract
Large Language Models (LLMs) are frequently characterized as universal learners, yet their ability to efficiently internalize fundamental physical laws from textual data remains a subject of debate. This paper introduces LEVERWORLDS, a generative framework designed to evaluate how models learn "world models" through physics-inspired tasks. Our analysis reveals that while Transformers can eventually master physical rules, they are significantly less sample-efficient than traditional statistical methods like Logistic Regression. We evaluate several learning paradigms, including fine-tuning and in-context learning, and propose a hybrid "Pipeline Learning" approach that utilizes LLMs as code architects to bridge the gap between natural language and structured statistical estimation.
Introduction
In machine learning, the bias-variance tradeoff dictates that models with strong inductive biases can learn from few samples, whereas flexible, low-bias models like Transformers require vast amounts of data to discover the same underlying structures. As LLMs are increasingly treated as general-purpose estimators, a critical question arises: can they efficiently learn simple physical worlds, such as the principles of torque, purely from text-based examples? This research investigates whether LLMs "discover" these rules or merely "brute-force" patterns through massive data exposure.
Methods
The authors introduce LEVERWORLDS, a framework that generates natural language descriptions of physical states involving a lever. The primary task is to predict whether a lever will tip Left (L) or Right (R) based on weights (mass), distances from the fulcrum, and their respective sides.
The study evaluates four distinct learning paradigms:
- Classic Baselines: Including Naive Maximum Likelihood Estimation (MLE) and Logistic Regression models of varying complexity.
- Transformer Fine-tuning: Training OPT models (125M to 6.7B parameters) using Low-Rank Adaptation (LoRA) on text-based physical descriptions.
- In-Context Learning (ICL): Prompting state-of-the-art models like GPT-4o with few-shot examples.
- Pipeline Learning: A hybrid approach where the LLM generates a Python parsing function to feed data into a classic statistical model.
Two experimental settings were established: World-1, where all variables are visible, and World-3, which introduces latent variables such as density and volume, requiring the model to infer underlying mass.
Results
The experimental results underscore a substantial "efficiency gap" between LLMs and traditional statistical methods. While fine-tuned OPT models eventually converge to low error rates, they require thousands of samples to achieve what simple polynomial models can master with a fraction of the data.
Key findings include:
- Sample Complexity: In the complex World-3 setting, Naive MLE requires approximately 10,000 samples to begin learning effectively, while Logistic Regression reaches high accuracy much faster.
- Structure Discovery: The sample inefficiency of LLMs is primarily attributed to the difficulty of discovering the underlying physical structure (the "physics") rather than modeling data variability.
- Pipeline Success: The proposed Pipeline Learning approach showed significant promise. By using an LLM (e.g., GPT-4o) to generate code that parses text into a structured format for a classic model, the system combines linguistic flexibility with statistical efficiency.
Conclusion
This research demonstrates that while LLMs are capable of learning physical world models, they lack the inductive bias necessary for sample-efficient rule discovery. The findings suggest that for tasks involving non-linguistic logic—such as physics or arithmetic—a modular AI architecture may be superior. In such a system, LLMs act as sophisticated interfaces or "architects" that delegate structured estimation tasks to specialized symbolic or statistical solvers.
Summary updated