Abstract
Traditional AI research suggests that Large Language Models (LLMs) require physical interaction or "grounding" to understand the physical world. This paper challenges that notion by introducing novel prompting techniques—specifically the "k-attribute" prompt—demonstrating that LLMs possess significant latent knowledge about object state changes during events. Our results show that language-only models can match the performance of models trained in simulated physical environments.
Introduction
In artificial intelligence, the "Frame Problem" describes the difficulty of modeling how actions affect the state of the world. Humans intuitively understand that throwing a laptop will likely break it and change its location. Previously, it was believed that LLMs, trained solely on text, lacked this grounded common sense. This study investigates whether the perceived performance gap between language-only models and physically-grounded models is a fundamental limitation or a result of suboptimal prompting.
Prompting Strategies for State Reasoning
The core of this research involves comparing different prompting strategies to evaluate how they unlock a model's ability to predict entity state changes:
- Zero-prompt (Baseline): The model predicts a fixed-dimension binary vector representing attribute changes from the context.
- Single-attribute prompt: The model is queried about one specific attribute at a time (e.g., "Is the location of the mug different?"), forcing focus on specific semantics.
- Multi-attribute prompt (all-attribute): The model receives a list of all possible attributes and identifies which ones changed.
- k-attribute prompt (Novel): Attributes are partitioned into random subsets during training. The model must determine which attributes within that specific subset changed. This prevents the model from memorizing training patterns and forces it to reason about the specific attributes requested.
Benchmarking Datasets
The researchers evaluated these strategies across two primary benchmarks that represent different levels of complexity:
- PiGLET: A dataset derived from the AI2-THOR simulator, featuring 5,000 examples across 14 physical attributes such as
isBrokenandtemperature. - OpenPI: A larger, more complex dataset based on real-world WikiHow articles. It includes 51 in-domain attributes and 40 out-of-domain (unseen) attributes, testing the model's ability to generalize to novel physical states.
Key Results and Insights
Performance Parity
On the PiGLET benchmark, a language-only RoBERTa model using these prompting techniques achieved a 93.9 F1 score, nearly matching the 94.4 F1 of the state-of-the-art "Physical Interaction" model. This suggests that the "grounding" provided by simulators may be redundant for sufficiently large language models when prompted correctly.
Generalization and Verbalization
Verbalizing attributes in the prompt acts as a semantic bridge, allowing models to tap into pre-trained world knowledge. The k-attribute method significantly improved performance on "out-of-domain" attributes by forcing the model to pay attention to the specific query rather than training biases. On the difficult OpenPI dataset, the T5 k-attribute model achieved an F1 of 51.4, a massive improvement over the previous GPT-2 baseline of 19.1.
Conclusion
EvEntS ReaLM demonstrates that LLMs already contain a wealth of "physical" common sense. By using structured prompting like the k-attribute method, we can bridge the gap between text-only training and real-world reasoning. While challenges remain regarding dataset noise and the computational cost of multi-pass inference, this work suggests that language itself may be a sufficient medium for learning the fundamental laws of cause and effect in the physical world.
Summary updated