JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions

Mo Yu, Yi Gu, Xiaoxiao Guo, Yufei Feng, Xiaodan Zhu, Michael A. Greenspan, Murray Campbell, Chuang Gan

Abstract

Current commonsense reasoning benchmarks often rely on static factual knowledge, which Large Language Models (LLMs) can frequently solve through memorization. This paper introduces two new benchmarks, JECC and ZUCC, derived from Interactive Fiction (IF) games. These tasks require "functional" commonsense—the ability to predict the outcomes of actions in a dynamic environment. Our evaluation reveals a significant performance gap between human experts and state-of-the-art models, including ChatGPT, highlighting the challenges of grounded reasoning in interactive worlds.

Introduction

Most existing commonsense reasoning benchmarks are "static," asking models to validate simple facts or identify social associations. However, these datasets suffer from two major flaws: the memorization shortcut, where LLMs "cheat" by recalling facts from pre-training, and a lack of dynamics. Real-world intelligence requires understanding how the world changes when we act upon it—a concept the authors term functional commonsense.

Example of interactive fiction gameplay requiring functional commonsense.

Interactive Fiction (IF) games, such as the iconic Zork, serve as an ideal environment for testing this reasoning. As shown in the example above, a player must understand that "climbing a tree" changes their elevation or that "hatching an egg" might reveal an item, while "burning repellent with a torch" could lead to a fatal explosion.

Methods: The Forward Prediction Task

The researchers frame the problem as a Forward Prediction Task. Given a current textual observation ($o_t$) and a chosen action ($a_t$), the model must select the most plausible next observation ($o_{t+1}$) from a set of candidates.

The Forward Prediction Task formulation.

To address this task, several neural architectures were implemented:

  • Pairwise Baselines: Standard models like Match LSTM and BERT-Siamese that compare the (observation + action) string against candidate outcomes.
  • Co-Matching Architectures: A sophisticated approach using bi-attention to jointly encode the relationship between the current state, the action, and the potential result. This allows the model to "cross-reference" specific words in the action (e.g., "climb") with elements in the observation (e.g., "tree").
  • Zero-Shot LLMs: ChatGPT was evaluated to determine if massive scale alone could solve these grounded logic puzzles.

Datasets: JECC and ZUCC

The paper introduces two primary benchmarks built from human gameplay walkthroughs:

  1. JECC (Jericho Environment Commonsense Comprehension): A large-scale dataset covering 29 games across genres like Sci-Fi, mystery, and fantasy, containing nearly 25,000 training tuples.
  2. ZUCC (Zork Universe Commonsense Comprehension): A smaller, focused dataset specifically using games from the iconic Zork series for faster model development.

Because these datasets are constructed from actual human paths through complex games, they represent natural reasoning challenges rather than artificial sentences.

Key Results and Insights

The results highlight a significant "commonsense gap":

  • Human Performance: Human experts achieved 92-96% accuracy.
  • Neural Models: The Co-Match BERT model was the top performer but only reached ~75% accuracy.
  • ChatGPT: In a zero-shot setting, ChatGPT struggled significantly, scoring only 51%.

The study found that the "BERT shortcut" is less effective here than in other NLP tasks, suggesting that the knowledge required to solve these games is not easily found in standard pre-training corpora. Furthermore, models must distinguish between descriptive "flavor text" and the actual state changes caused by an action.

Conclusion

By releasing JECC and ZUCC, the researchers provide a new yardstick for measuring whether AI is truly learning how the world works. The significant gap between human and model performance suggests that functional, grounded logic remains a major frontier for artificial intelligence.

Summary updated

Abstract

Commonsense reasoning simulates the human ability to make presumptions about our physical world, and it is an essential cornerstone in building general AI systems. We proposea new commonsense reasoning dataset based on human’s Interactive Fiction (IF) gameplaywalkthroughs as human players demonstrate plentiful and diverse commonsense reasoning. The new dataset provides a natural mixture of various reasoning types and requires multi-hopreasoning. Moreover, the IF game-based construction procedure requires much less humaninterventions than previous ones. Different from existing benchmarks, our dataset focuseson the assessment of functional commonsense knowledge rules rather than factual knowledge. Hence, in order to achieve higher performance on our tasks, models need to effectively uti-lize such functional knowledge to infer the outcomes of actions, rather than relying solely onmemorizing facts. Experiments show that the introduced dataset is challenging to previousmachine reading models as well as the new large language models with a significant 20%performance gap compared to human experts.
BibTeX
@inproceedings{yu2023jecc,
  author = {Mo Yu and Yi Gu and Xiaoxiao Guo and Yufei Feng and Xiaodan Zhu and Michael A. Greenspan and Murray Campbell and Chuang Gan},
  title = {{JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.713},
}
Figure