JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions

Mo Yu, Xiaoxiao Guo, Yufei Feng, Yi Gu, Xiaodan Zhu, Michael A. Greenspan, Murray Campbell, Chuang Gan

Abstract

Commonsense reasoning simulates the human ability to make presumptions about our physical world, and it is an essential cornerstone in building general AI systems. We propose a new commonsense reasoning dataset based on human's Interactive Fiction (IF) gameplay walkthroughs as human players demonstrate plentiful and diverse commonsense reasoning. The new dataset provides a natural mixture of various reasoning types and requires multi-hop reasoning. Moreover, the IF game-based construction procedure requires much less human interventions than previous ones. Different from existing benchmarks, our dataset focuses on the assessment of functional commonsense knowledge rules rather than factual knowledge. Hence, in order to achieve higher performance on our tasks, models need to effectively utilize such functional knowledge to infer the outcomes of actions, rather than relying solely on memorizing facts. Experiments show that the introduced dataset is challenging to previous machine reading models as well as the new large language models with a significant 20% performance gap compared to human experts.

Introduction

Traditional commonsense reasoning benchmarks often focus on factual knowledge—static associations like "a bird has wings." However, large language models (LLMs) frequently "cheat" on these tasks by recalling patterns from their massive training corpora without performing actual reasoning. True human commonsense is functional: it involves understanding the dynamic consequences of actions within a world model.

This paper introduces two new benchmarks, JECC (Jericho Environment Commonsense Comprehension) and ZUCC (Zork Universe Commonsense Comprehension), derived from Interactive Fiction (IF). These text-based adventure games require players to navigate complex environments, interact with objects, and solve puzzles using grounded physical and spatial logic.

Task Formulation and Datasets

We frame commonsense reasoning as a Forward Prediction Task. Given a textual observation of the current state ($o_t$) and a chosen action ($a_t$), the model must select the correct next observation ($o_{t+1}$) from a set of candidates. This requires the model to simulate the outcome of an action based on the narrative context.

The Forward Prediction Task: Models must select the correct next observation given a current state and action.

The datasets were constructed by extracting human "walkthroughs" from the Jericho environment, ensuring the reasoning required is purposeful.

  • JECC: A large-scale dataset covering 29 games across genres like Sci-Fi and Mystery, containing approximately 25,000 processed tuples.
  • ZUCC: A specialized dataset focused on the Zork universe for domain-specific testing.

Methodology

To establish baselines, we evaluated several neural architectures:

  • Pairwise Models: Standard BERT and LSTM models that evaluate the (observation + action) against candidate next states.
  • Co-Matching Models: Advanced architectures that use bi-attention to explicitly model the interactions between the current state, the action, and the potential outcome. This approach better mimics how humans weigh an action against a predicted result.

Key Results

Our experiments highlight a significant "human-machine gap." While human experts achieve accuracy scores between 92% and 96%, the best-performing supervised model (Co-Match BERT) only reaches approximately 75%.

Performance comparison of various models on the ZUCC and JECC benchmarks, highlighting the gap between AI and human performance.

Notably, ChatGPT (zero-shot) performed poorly, scoring only 51% on a subset of JECC. This suggests that even powerful LLMs struggle with grounded, functional logic when not specifically trained for interactive environments. Furthermore, the relatively small performance boost provided by BERT over simpler LSTMs—compared to traditional benchmarks—indicates that JECC and ZUCC are more resistant to simple memorization strategies.

Conclusion

JECC and ZUCC represent a significant step toward evaluating "world model" reasoning in AI. By leveraging the rich narratives of Interactive Fiction, we provide a benchmark that requires multi-hop, functional logic rather than static fact retrieval. Future work will explore incorporating longer-term history and more complex action spaces to further challenge and develop general AI systems.

Summary updated

Abstract

Commonsense reasoning simulates the human ability to make presumptions about our physical world, and it is an essential cornerstone in building general AI systems. We propose a new commonsense reasoning dataset based on human's Interactive Fiction (IF) gameplay walkthroughs as human players demonstrate plentiful and diverse commonsense reasoning. The new dataset provides a natural mixture of various reasoning types and requires multi-hop reasoning. Moreover, the IF game-based construction procedure requires much less human interventions than previous ones. Different from existing benchmarks, our dataset focuses on the assessment of functional commonsense knowledge rules rather than factual knowledge. Hence, in order to achieve higher performance on our tasks, models need to effectively utilize such functional knowledge to infer the outcomes of actions, rather than relying solely on memorizing facts. Experiments show that the introduced dataset is challenging to previous machine reading models as well as the new large language models with a significant 20% performance gap compared to human experts.
BibTeX
@inproceedings{yu2023jecc,
  author = {Mo Yu and Xiaoxiao Guo and Yufei Feng and Yi Gu and Xiaodan Zhu and Michael A. Greenspan and Murray Campbell and Chuang Gan},
  title = {{JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.713},
}
Figure