Data Augmentation for Learning to Play in Text-Based Games

Jinhyeon Kim, Kee-Eung KimIJCAI 2022 · DOI

Abstract

Reinforcement learning (RL) agents often struggle to generalize in text-based games, frequently overfitting to specific training scenarios rather than learning underlying game logic. We introduce Transition-Matching Permutation (TMP), a data augmentation technique that identifies and swaps linguistically symmetric phrases—such as "carrot" and "banana"—that share identical game dynamics. By forcing agents to learn functional roles rather than memorizing names, TMP enables significant improvements in generalization, achieving state-of-the-art results on the Cooking Game benchmark.

Introduction: The Challenge of Generalization in Text RL

In text-based games, agents interact with their environment through natural language. A common failure mode is "overfitting" to training levels. For instance, if a training set always places a carrot in the fridge, an agent might learn a hardcoded rule like "go to the fridge for the carrot," failing when a test level places a banana there instead.

While computer vision-based RL benefits from standard augmentations like flipping or cropping, text-based environments lack a clear equivalent. The primary challenge is modifying text without violating the game's internal logic or "dynamics."

Transition-Matching Permutation (TMP)

The proposed TMP method discovers and exploits symmetries within the game world through a three-step pipeline:

  1. Candidate Phrase Extraction: Using the SpaCy NLP library, the system extracts noun phrases from pre-collected game trajectories, focusing on objects found in action commands (e.g., "apple" from "take apple").
  2. Filtering via Trajectory Data: The system determines if two phrases are interchangeable by calculating a matching score. If swapping "carrot" for "banana" in an action sequence consistently results in equivalent observations and rewards, the pair is deemed symmetric.
  3. Training with Permutations: During training, the agent randomly selects a valid permutation at the start of each episode. It perceives "permuted" observations and must issue "permuted" actions, effectively decoupling the object's name from its functional role.
Overview of the Transition-Matching Permutation (TMP) process, showing how trajectory data is used to generate augmented environments.

Identifying Linguistic Symmetries

The core innovation lies in identifying which words can be swapped without breaking the game. By analyzing transition data, the model can cluster objects that behave identically within the game engine. This allows the system to generate diverse training scenarios that remain logically consistent.

Visualization of discovered symmetries: phrases like food items or types of doors cluster together based on their interchangeable roles in the game dynamics.

Experimental Setup and Results

The method was evaluated using the TextWorld framework, specifically the Cooking Game suite—a benchmark of four levels with increasing complexity involving navigation, ingredient collection, and recipe preparation.

Key Performance Gains

The addition of TMP led to substantial performance improvements:

  • State-of-the-Art Results: Combining TMP with observation pruning (CREST-cat) set new records on the benchmark.
  • Significant Score Boosts: On Level 2, the normalized score increased from 40.4 to 75.5, representing an 87% improvement.
  • Behavioral Shift: Agents trained with TMP showed a dramatic increase in the "Read" metric (from 6% to 86% on Level 2). This indicates the agent learned to actually read the cookbook to find the recipe, rather than relying on memorized ingredient locations.

Conclusion and Future Work

TMP demonstrates that linguistic symmetry can serve as a powerful regularizer for text-based RL, analogous to spatial symmetry in visual domains. While currently limited to 1-to-1 phrase swaps, future work could extend this to more complex n-tuple permutations or integrate it with agents that maintain explicit belief states. By forcing agents to focus on functional roles, TMP provides a robust path toward more generalizable language-based AI.

Summary updated

BibTeX
@inproceedings{kim2022data,
  author = {Jinhyeon Kim and Kee-Eung Kim},
  title = {{Data Augmentation for Learning to Play in Text-Based Games}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/436},
}
Figure