JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions

Mo Yu, Yi Gu, Xiaoxiao Guo, Yufei Feng, Xiaodan Zhu, Michael A. Greenspan, Murray Campbell, Chuang Gan

Abstract

Commonsense reasoning simulates the human ability to make presumptions about our physical world, and it is an essential cornerstone in building general AI systems. We proposea new commonsense reasoning dataset based on human’s Interactive Fiction (IF) gameplaywalkthroughs as human players demonstrate plentiful and diverse commonsense reasoning. The new dataset provides a natural mixture of various reasoning types and requires multi-hopreasoning. Moreover, the IF game-based construction procedure requires much less humaninterventions than previous ones. Different from existing benchmarks, our dataset focuseson the assessment of functional commonsense knowledge rules rather than factual knowledge. Hence, in order to achieve higher performance on our tasks, models need to effectively uti-lize such functional knowledge to infer the outcomes of actions, rather than relying solely onmemorizing facts. Experiments show that the introduced dataset is challenging to previousmachine reading models as well as the new large language models with a significant 20%performance gap compared to human experts.
BibTeX
@inproceedings{yu2023jecc,
  author = {Mo Yu and Yi Gu and Xiaoxiao Guo and Yufei Feng and Xiaodan Zhu and Michael A. Greenspan and Murray Campbell and Chuang Gan},
  title = {{JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.713},
}
Figure