TL;DR
This paper introduces JECC and ZUCC, two new benchmarks for commonsense reasoning derived from interactive fiction (IF) game walkthroughs. These datasets challenge models to predict the next observation given a game state and an action, requiring multi-hop reasoning and functional knowledge beyond simple fact retrieval, and reveal a significant performance gap compared to human experts.
What Problem Does This Paper Address?
Building AI systems that can reason about the physical world, akin to humans, is a long-standing goal. While numerous datasets exist for commonsense reasoning, they often suffer from limitations: they tend to focus on single reasoning types, rely on validation-based tasks (e.g., choosing the correct statement), and can be solved by models that have memorized factual knowledge rather than truly understanding functional rules. This makes it difficult to evaluate models on their ability to perform complex, multi-hop reasoning in dynamic environments. The authors aim to address this by creating a dataset that requires a deeper understanding of how actions affect the world and predicting subsequent states, mimicking the challenges faced in interactive text-based games.
Methods
The core of this work is the creation of new datasets, JECC and ZUCC, by leveraging gameplay data from interactive fiction (IF) games. The authors extract "walkthroughs" – sequences of observations and actions that lead to successful game completion. They then formulate the task as a forward prediction problem: given a current observation (game state) and an action, predict the most likely next observation. This approach naturally incorporates multi-hop reasoning and requires understanding functional knowledge (e.g., how an action causes an effect) rather than just factual knowledge. They also propose and evaluate several neural inference baselines, including:
- Pairwise Textual Inference: Match LSTM, BERT Siamese, and BERT Concat, which treat the problem as predicting a hypothesis (next observation) given a premise (current observation + action).
- Triplewise Inference: Co-Matching LSTM and Co-Matching BERT, which explicitly model the interaction between the observation, action, and candidate next observation, aiming for better reasoning capabilities.
- Large Language Models: They also test ChatGPT in a zero-shot setting to gauge the capabilities of current state-of-the-art LLMs.
Datasets
- JECC (Jericho Environment Commonsense Comprehension): Derived from 29 games in the Jericho Environment, this is the larger dataset. It focuses on evaluating models on tuples from actual walkthroughs to ensure a natural distribution of commonsense knowledge.
- ZUCC (Zork Universe Commonsense Comprehension): Derived from four IF games in the Zork Universe, this smaller dataset is intended for faster model development and testing.
These datasets differ from existing ones by:
- Requiring multi-hop reasoning.
- Focusing on functional commonsense knowledge (rules) rather than factual knowledge.
- Being constructed from natural gameplay, leading to a more diverse and less biased distribution of reasoning types.
- Using a forward prediction task (predicting the next state) rather than simple validation.
Key Insights and Ideas
- IF games as a rich source of commonsense: The authors highlight that IF games, through their narrative structure and gameplay mechanics, naturally embed a wide range of commonsense knowledge, including spatial reasoning, object interactions, and physical causality.
- Functional knowledge is key: The proposed tasks are designed to move beyond simple fact retrieval. Models need to understand how actions lead to outcomes, not just what the outcomes are. This is crucial for building more robust AI.
- Co-matching architectures excel: The triplewise inference models (Co-Match LSTM and Co-Match BERT) consistently outperform pairwise models, suggesting that explicitly modeling the interaction between observation, action, and candidate next observation is beneficial for this task.
- LLMs still struggle with deep reasoning: Despite their general capabilities, even advanced LLMs like ChatGPT show poor performance on this task, indicating that current prompting techniques might not be sufficient for complex, multi-hop commonsense reasoning in dynamic environments.
Key Results
- Significant Human-Machine Gap: The best performing models (Co-Match BERT) achieve accuracies around 75% on JECC, while human experts reach over 92%. This substantial gap (around 20%) underscores the difficulty of the tasks and the limitations of current models.
- Co-matching outperforms pairwise: Co-Match BERT achieves the highest accuracy (75.48% on JECC test) among the evaluated models, outperforming BERT Siamese and BERT Concat.
- BERT's limitations with long sequences: BERT-concat struggles with longer input sequences in JECC, suggesting potential issues with its fixed input length.
- Human evaluation reveals nuances: Human evaluators identified cases where even they lacked sufficient clues, providing an upper bound for model performance without access to historical context.
Limitations and Future Work
- Knowledge Distribution: The authors acknowledge that achieving a perfectly natural distribution of commonsense knowledge types is challenging and suggest future work involving human annotation to refine this.
- Fairness Considerations: The reliance on text games written at a specific time might introduce biases. The authors propose that using a wider variety of games and better sampling strategies can mitigate this.
- LLM Prompting: For future work, they suggest exploring advanced prompting techniques like Chain-of-Thought for LLMs to improve their performance on these reasoning-intensive tasks.
- Dataset Expansion: The IF game-based construction allows for easy expansion of the dataset by including more games and increasing the prediction horizon (multi-step actions).
Summary updated