DiffG-RL: Leveraging Difference between Environment State and Common Sense

Tsunehiko Tanaka, Daiki Kimura, Michiaki TatsuboriEMNLP 2022 · DOI

Introduction

Reinforcement Learning (RL) agents in text-based games face a unique challenge: navigating partially observable environments using natural language. Success requires more than a policy; it requires common sense. While agents can be equipped with external knowledge bases, they often struggle with information overload or failing to apply that knowledge effectively to specific tasks.

Conceptual overview of DiffG-RL: bridging the gap between observation and common sense.

Researchers from Waseda University and IBM Research have introduced DiffG-RL, an agent that explicitly models the discrepancy between the current environment state and "ideal" common sense configurations. By focusing on this "difference," the agent can more effectively prioritize actions that move the world toward a tidy state.

Methodology

The DiffG-RL architecture is built upon four primary components designed to extract, compare, and act upon environmental and external knowledge:

  1. Environment State Extractor (EE): Utilizes an Abstract Meaning Representation (AMR) parser to identify the current state of objects (e.g., "dirty fork on the floor").
  2. Common Sense Extractor (CE): Employs semantic embeddings (GloVe) to retrieve relevant triples from the Visual Genome (VG) dataset, identifying where objects "should" be (e.g., "fork in dishwasher").
  3. Difference Encoder (DE): This core innovation constructs a graph where object nodes are linked to both their observed state and their common sense "ideal" state. A Graph Isomorphism Network (GIN) encodes the resulting "difference graph."
  4. Action Selector (AS): Scores admissible commands by integrating the encoded differences, allowing the agent to select the action that best resolves the observed discrepancy.

Experimental Results

The model was evaluated using the TextWorld Commonsense (TWC) benchmark across varying difficulty levels. A critical test of the agent's capability was its performance on "OUT" sets—scenarios involving objects and locations not seen during training.

Performance comparison of DiffG-RL against state-of-the-art baselines on the TWC benchmark.

DiffG-RL demonstrated a significant performance leap, achieving a 17% average improvement over previous state-of-the-art models. Notably, on the "Hard" difficulty setting with unseen entities, the model improved scores by 40%. These results suggest that modeling the difference between states is a more robust approach to generalization than simple knowledge retrieval.

Discussion and Conclusion

The success of DiffG-RL highlights the importance of "dissonance-driven" action selection. By identifying where the environment deviates from common sense, the agent avoids the "looping" behaviors common in standard RL agents.

Future work aims to expand the model's capability to handle multi-object relationships and integrate more advanced Transformer-based embeddings to further refine semantic matching. Ultimately, DiffG-RL provides a scalable framework for grounding RL agents in human-like common sense.

Reference: Tanaka, T., Kimura, D., & Tatsubori, M. (2022). DiffG-RL: Leveraging Difference between State and Common Sense. Findings of EMNLP 2022.

Summary updated

BibTeX
@inproceedings{tanaka2022diffg,
  author = {Tsunehiko Tanaka and Daiki Kimura and Michiaki Tatsubori},
  title = {{DiffG-RL: Leveraging Difference between Environment State and Common Sense}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.110},
}
Figure