Learning Object-Oriented Dynamics for Planning from Text

Guiliang Liu, Ashutosh Adhikari, Amir-massoud Farahmand, Pascal PoupartICLR 2022 · OpenReview

TL;DR

This paper introduces Object-Oriented Text Dynamics (OOTD), a model-based reinforcement learning approach designed specifically for text-based games. By factorizing the world into a persistent memory graph of objects and using modular transition layers to update them, OOTD allows agents to plan effectively in complex, partially observable text environments, significantly outperforming traditional model-free baselines.

The Challenge of Text-Based Games

Text-Based Games (TBGs)—such as digital "choose your own adventure" or classic titles like Zork—present a unique challenge for artificial intelligence. Unlike image-based games where the state is often fully visible, TBGs are partially observable and high-dimensional. An observation might be a long paragraph of text, but only a tiny fraction of it (e.g., "the carrot is on the counter") is actually relevant to the underlying game state.

Most existing RL agents for these games are "model-free," meaning they learn a direct mapping from text to actions. However, these agents struggle with sample efficiency and generalization. To use "model-based" planning—where an agent imagines future outcomes before acting—the agent needs a dynamics model that can predict how the world changes. Building such a model for text is difficult because language is noisy, and the model must "remember" objects that aren't explicitly mentioned in the current turn.

Object-Oriented Text Dynamics (OOTD)

The authors propose the OOTD model, which treats the game as an Object-Oriented Partially Observable Markov Decision Process (OO-POMDP).

OOTD Architecture and Planning

Key components of this approach include:

  • Memory Graph as an Information Bottleneck: Instead of processing raw text every turn, OOTD maintains a knowledge graph where nodes represent objects. This graph acts as a "memory," filtering out irrelevant "flavor text" and keeping track of objects seen in previous rooms.
  • Action-Aware Representations (BiDAF): To understand which objects an action affects, the model uses a Bi-Directional Attention Flow (BiDAF) network. If the agent says "take the carrot," the model uses attention to focus specifically on the "carrot" and "counter" nodes in its graph.
  • Independent Transition Layers: A major innovation is the use of modular, independently parameterized layers for each object. When an action is taken, only the affected objects' states are updated, while others remain invariant. This mimics the causal structure of the real world.
  • Variational Learning (ELBo): The model is trained using variational objectives. The authors offer two flavors: OS-ELBo (Object-Supervised), which uses ground-truth graph data for training, and SS-ELBo (Self-Supervised), which learns directly from text observations and rewards, making it more applicable to environments where ground-truth graphs aren't available.

Key Insights

  • The Power of Factorization: By breaking the world into discrete objects rather than one giant "state vector," the model becomes much more robust. If the agent learns how a "carrot" behaves, that knowledge generalizes even if the carrot is in a different room or a different game.
  • Filtering the Noise: The memory graph serves as a crucial filter. By forcing the dynamics model to predict graph updates rather than raw text, the agent ignores the "noisy" parts of language that don't affect gameplay.
  • The "Objective Mismatch" Phenomenon: Interestingly, the self-supervised (SS) model often performed better at controlling the agent than the object-supervised (OS) model. The authors suggest this is because the SS model focuses specifically on reward-relevant features, whereas the OS model might spend capacity tracking details that don't actually help win the game.

Results and Performance

OOTD was evaluated on TextWorld, a widely used framework for generating synthetic text-based games, specifically using cooking-themed games across five difficulty levels.

Training curves demonstrating OOTD sample efficiency
  • Superior Performance: OOTD-based planners (using Dyna-Q or Monte Carlo Tree Search) significantly outperformed state-of-the-art model-free baselines like DRQN and GATA. On the hardest difficulty levels, the OOTD agent achieved scores nearly double those of the best baselines.
  • Sample Efficiency: Because OOTD can "imagine" transitions, it learns much faster. The training curves show OOTD reaching high performance levels with far fewer interactions with the actual game environment.
  • Generalization: The model showed a strong ability to generalize to unseen games in the test set, proving that the object-oriented approach captures the underlying "physics" of the text world rather than just memorizing strings.

Limitations and Future Work

The current study focuses on "choice-based" games where the agent selects from a list of valid commands. In "parser-based" games, the agent must generate text character-by-character, which is a much larger action space. Furthermore, while successful in the structured world of TextWorld, scaling this to highly creative or metaphorical interactive fiction would require even more sophisticated language grounding. Finally, improving the "cold start" performance of graph discovery remains a challenge for fully self-supervised deployments.

Summary updated

Abstract

The advancement of dynamics models enables model-based planning in complex environments. Existing dynamics models commonly study image-based games with fully observable states. Generalizing these models to Text-Based Games (TBGs), which commonly describe the partially observable states with noisy text observations, is challenging. In this work, we propose an Object-Oriented Text Dynamics (OOTD) model that enables planning algorithms to solve decision-making problems in text domains. OOTD predicts a memory graph that dynamically remembers the history of object observations and filters object-irrelevant information. To facilitate the robustness of dynamics, our OOTD model identifies the objects influenced by input actions and predicts the belief of object states with independently parameterized transition layers. We develop variational objectives under the object-supervised and self-supervised settings to model the stochasticity of predicted dynamics. Empirical results show OOTD-based planner significantly outperforms model-free baselines in terms of sample efficiency and running scores.
BibTeX
@inproceedings{liu2022learning,
  author = {Guiliang Liu and Ashutosh Adhikari and Amir-massoud Farahmand and Pascal Poupart},
  title = {{Learning Object-Oriented Dynamics for Planning from Text}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=B6EIcyp-Rb7},
}
Figure