Abstract
Current commonsense reasoning benchmarks often rely on static factual knowledge, which Large Language Models (LLMs) can frequently solve through memorization. This paper introduces two new benchmarks, JECC and ZUCC, derived from Interactive Fiction (IF) games. These tasks require "functional" commonsense—the ability to predict the outcomes of actions in a dynamic environment. Our evaluation reveals a significant performance gap between human experts and state-of-the-art models, including ChatGPT, highlighting the challenges of grounded reasoning in interactive worlds.
Introduction
Most existing commonsense reasoning benchmarks are "static," asking models to validate simple facts or identify social associations. However, these datasets suffer from two major flaws: the memorization shortcut, where LLMs "cheat" by recalling facts from pre-training, and a lack of dynamics. Real-world intelligence requires understanding how the world changes when we act upon it—a concept the authors term functional commonsense.
Interactive Fiction (IF) games, such as the iconic Zork, serve as an ideal environment for testing this reasoning. As shown in the example above, a player must understand that "climbing a tree" changes their elevation or that "hatching an egg" might reveal an item, while "burning repellent with a torch" could lead to a fatal explosion.
Methods: The Forward Prediction Task
The researchers frame the problem as a Forward Prediction Task. Given a current textual observation ($o_t$) and a chosen action ($a_t$), the model must select the most plausible next observation ($o_{t+1}$) from a set of candidates.
To address this task, several neural architectures were implemented:
- Pairwise Baselines: Standard models like Match LSTM and BERT-Siamese that compare the (observation + action) string against candidate outcomes.
- Co-Matching Architectures: A sophisticated approach using bi-attention to jointly encode the relationship between the current state, the action, and the potential result. This allows the model to "cross-reference" specific words in the action (e.g., "climb") with elements in the observation (e.g., "tree").
- Zero-Shot LLMs: ChatGPT was evaluated to determine if massive scale alone could solve these grounded logic puzzles.
Datasets: JECC and ZUCC
The paper introduces two primary benchmarks built from human gameplay walkthroughs:
- JECC (Jericho Environment Commonsense Comprehension): A large-scale dataset covering 29 games across genres like Sci-Fi, mystery, and fantasy, containing nearly 25,000 training tuples.
- ZUCC (Zork Universe Commonsense Comprehension): A smaller, focused dataset specifically using games from the iconic Zork series for faster model development.
Because these datasets are constructed from actual human paths through complex games, they represent natural reasoning challenges rather than artificial sentences.
Key Results and Insights
The results highlight a significant "commonsense gap":
- Human Performance: Human experts achieved 92-96% accuracy.
- Neural Models: The Co-Match BERT model was the top performer but only reached ~75% accuracy.
- ChatGPT: In a zero-shot setting, ChatGPT struggled significantly, scoring only 51%.
The study found that the "BERT shortcut" is less effective here than in other NLP tasks, suggesting that the knowledge required to solve these games is not easily found in standard pre-training corpora. Furthermore, models must distinguish between descriptive "flavor text" and the actual state changes caused by an action.
Conclusion
By releasing JECC and ZUCC, the researchers provide a new yardstick for measuring whether AI is truly learning how the world works. The significant gap between human and model performance suggests that functional, grounded logic remains a major frontier for artificial intelligence.
Summary updated