EvEntS ReaLM: Event Reasoning of Entity States via Language Models

Evangelia Spiliopoulou, Artidoro Pagnoni, Yonatan Bisk, Eduard H. Hovy

Abstract

This paper investigates models of event implications. Specifically, how well models predict entity state-changes, by targeting their understanding of physical attributes. Nominally, Large Language models (LLM) have been exposed to procedural knowledge about how objects interact, yet our benchmarking shows they fail to reason about the world. Conversely, we also demonstrate that existing approaches often misrepresent the surprising abilities of LLMs via improper task encodings and that proper model prompting can dramatically improve performance of reported baseline results across multiple tasks. In particular, our results indicate that our prompting technique is especially useful for unseen attributes (out-of-domain) or when only limited data is available.

Introduction

When an agent interacts with an object—for instance, dropping a glass mug—humans can intuitively predict the resulting state changes: the mug will likely break, its location will change to the floor, and it will no longer be "held." In artificial intelligence, this ability to model the effects of actions on entities is known as the Frame Problem.

A long-standing hypothesis in AI research suggests that true "common sense" regarding physical reality requires grounded experience—actual interaction with a physical environment. Recent studies have argued that language-only models are fundamentally limited in this domain. However, "EvEntS ReaLM" challenges this notion, investigating whether these perceived limitations are inherent to Large Language Models (LLMs) or simply a byproduct of how we prompt them to reason.

Methodology: Rethinking Task Encoding

The core contribution of this work is a systematic study of task encoding. Rather than treating state-change prediction as a "black box" multi-label classification problem where the model outputs a binary vector, the authors propose "verbalizing" the task. They evaluate four distinct prompting strategies:

  1. Zero-prompt (Baseline): The model receives the context and is expected to predict a binary vector of changes without explicit textual descriptions of the attributes.
  2. Single-attribute Prompt: The model is queried for each attribute individually (e.g., "Is the location of the mug different?"). This forces the model to isolate and focus on one physical property at a time.
  3. Multi-attribute (All-attribute) Prompt: The model is provided with a comprehensive list of all possible attributes and generates text describing which ones changed.
  4. $k$-attribute Prompt (Novel): The model is prompted with a random subset of $k$ attributes. This technique prevents the model from "lazily" memorizing a fixed output structure and forces it to attend to the specific semantic meaning of the attributes provided in the prompt.

These strategies were tested across three model architectures: RoBERTa (encoder-only), T5 (encoder-decoder), and GPT-3 (decoder-only).

Experimental Framework

The researchers utilized two primary benchmarks to evaluate physical reasoning:

  • PiGLET: A semi-artificial dataset derived from the AI2-THOR physical simulator, featuring 5,000 examples across 14 physical attributes.
  • OpenPI: A complex dataset based on real-world WikiHow articles. It includes 51 "in-domain" attributes and approximately 40 "out-of-domain" attributes—properties that appear in the test set but were never seen during training.

Key Insights and Results

The study yielded several surprising insights into how LLMs process physical information:

1. Interaction is Not Always Mandatory

On the PiGLET benchmark, language-only models using optimized prompts achieved an F1 score of 94.1, nearly identical to models specifically trained with physical interaction data (94.4 F1). This suggests that the vast amount of procedural knowledge in text corpora may be sufficient for many reasoning tasks.

2. Verbalization as a Semantic Bottleneck

Naming the attribute (e.g., "location," "temperature") in the prompt acts as a semantic bridge. It allows the model to map its internal pre-trained knowledge to the specific task at hand, significantly improving performance over abstract vector outputs.

3. Generalization via Randomization

The $k$-attribute prompt proved essential for generalization. By varying the attributes shown during training, the model learned the relationship between the name of an attribute and the context of the action. This was most evident in the OpenPI Out-of-Domain results: while the zero-prompt baseline cannot predict unseen attributes, the RoBERTa single-attribute prompt reached an F1 of 23.5 for attributes it had never encountered in training.

4. Hidden Model Competence

A manual audit of the models' "errors" revealed that nearly 50% of the flagged mistakes were actually false negatives. The models often correctly predicted a state change that human annotators had overlooked, suggesting that LLMs may be more capable than standard benchmarks indicate.

Limitations and Future Directions

Despite the success of these prompting techniques, several challenges remain:

  • Inference Cost: The single-attribute approach requires multiple passes (one per attribute), which is computationally expensive for large attribute sets.
  • Annotation Noise: The OpenPI dataset contains significant inconsistencies, which likely limits the measurable performance of any model.
  • Complex Reasoning: While LLMs excel at simple state changes, multi-step physical reasoning in uncontrolled environments remains an open problem.

Conclusion

EvEntS ReaLM demonstrates that the perceived gap in LLM physical reasoning is often a matter of communication rather than capability. By employing $k$-attribute prompting and verbalized task encodings, we can unlock significant "common sense" reasoning about the physical world, even for attributes the models have never explicitly been trained to evaluate.

Summary updated

Abstract

This paper investigates models of event implications. Specifically, how well models predict entity state-changes, by targeting their understanding of physical attributes. Nominally, Large Language models (LLM) have been exposed to procedural knowledge about how objects interact, yet our benchmarking shows they fail to reason about the world. Conversely, we also demonstrate that existing approaches often misrepresent the surprising abilities of LLMs via improper task encodings and that proper model prompting can dramatically improve performance of reported baseline results across multiple tasks. In particular, our results indicate that our prompting technique is especially useful for unseen attributes (out-of-domain) or when only limited data is available.
BibTeX
@inproceedings{spiliopoulou2022events,
  author = {Evangelia Spiliopoulou and Artidoro Pagnoni and Yonatan Bisk and Eduard H. Hovy},
  title = {{EvEntS ReaLM: Event Reasoning of Entity States via Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=hIrojI8qahi},
}
Figure