VisualHints: A Visual-Lingual Environment for Multimodal Reinforcement Learning

Thomas Carta, Subhajit Chaudhury, Kartik Talamadupula, Michiaki TatsuboriarXiv 2020 · arXiv

Abstract

Reinforcement Learning (RL) agents typically operate within single modalities, such as pixel-based visual inputs or text-based natural language. However, real-world tasks often require the simultaneous integration of diverse information sources. We introduce VisualHints, a novel multimodal RL environment that extends text-based games with visual floor plans. By requiring agents to "examine" visual hints to resolve textual ambiguities and avoid hazards, VisualHints provides a benchmark for spatial reasoning and multimodal fusion. We present a baseline architecture utilizing CNNs and LSTMs and evaluate its performance against state-of-the-art unimodal models, highlighting the unique challenges of cross-modal integration.

Introduction

Traditional RL environments often isolate modalities. Text-based games (TBGs) are effective for testing language understanding and long-term planning but suffer from partial observability, where the agent's knowledge is restricted to the current room's description. In contrast, human intelligence frequently relies on visual context—such as a map—to navigate complex environments and interpret verbal instructions.

Overview of the VisualHints environment, where an agent must integrate textual room descriptions with visual floor plans to navigate and complete tasks.

VisualHints addresses this gap by modifying Microsoft’s TextWorld "Cooking Game" to include visual "hints." These hints are floor plans that the agent must utilize to navigate efficiently and avoid "Death Rooms"—high-penalty zones that lead to immediate failure. This setup forces the agent out of unimodal silos, requiring true multimodal synthesis to reach optimal solutions.

The VisualHints Environment

VisualHints is built as an extension layer over TextWorld, offering a customizable benchmark for researchers. The environment generates floor plans with varying difficulty levels and features that can be toggled to test specific agent capabilities:

  • Death Rooms: Zones that impose heavy penalties, necessitating visual identification to avoid.
  • Color Paths: Visual guides that can be used to simplify navigation.
  • Room Importance: A feature that masks room names and replaces them with numerical priorities, testing the agent's ability to reason about abstract spatial data.

The environment includes evaluation sets for "Navigation Games" (ranging from 6 to 12 rooms) and "Non-Navigation Games" where the agent begins in the target kitchen.

Multimodal Agent Architecture

The proposed baseline agent employs an Advantage Actor-Critic (A2C) framework designed for multimodal fusion. The architecture consists of three primary encoding modules:

  1. Textual Module: Encodes game state descriptions and available natural language commands using an LSTM.
  2. Visual Module: Utilizes a CNN to extract features from floor plans. These features are processed by a bidirectional LSTM to capture spatial relationships and the global topology of the map.
  3. Fusion & Policy: Features from both modalities are concatenated. This fused representation is used by the policy network to select actions and by the value network to estimate the potential reward of the current state.
The multimodal agent architecture, featuring separate encoding modules for images, context, and commands, fused for value estimation and action selection.

To ensure the visual module effectively interprets spatial data, it was pre-trained on various binary classification tasks, such as identifying room types or counting inaccessible rooms.

Experimental Results and Discussion

The multimodal baseline was compared against LeDeepChef, a state-of-the-art text-only model. In complex 12-room navigation tasks, the Visual-Lingual model achieved a success rate of 0.13 on the test set, compared to 0.31 for LeDeepChef.

While the multimodal baseline did not outperform the highly optimized text-only SOTA, these results establish a critical baseline for a more difficult problem class. Analysis of the agent's behavior provided several insights:

  • Spatial Reasoning Bottleneck: Pre-training revealed that while agents easily identify room existence (0.99 F1), they struggle with topological reasoning, such as counting rooms blocked by hazards (0.8 F1). This indicates that "seeing" a map is distinct from "understanding" its spatial implications.
  • Visual Attention: CNN filter analysis showed the model successfully learned to highlight specific rooms and identify the "death room" based on color and position.

Conclusion

VisualHints provides a necessary sandbox for developing agents capable of simultaneous visual and lingual reasoning. The primary challenge remains the effective fusion of disparate data types—pixels and words—within a massive natural language action space. Future work will explore more sophisticated fusion mechanisms, such as cross-modal attention, and evaluate the environment with advanced RL algorithms beyond A2C.

Summary updated

Abstract

We present VisualHints, a novel environment for multimodal reinforcement learning (RL) involving text-based interactions along with visual hints (obtained from the environment). Real-life problems often demand that agents interact with the environment using both natural language information and visual perception towards solving a goal. However, most traditional RL environments either solve pure vision-based tasks like Atari games or video-based robotic manipulation; or entirely use natural language as a mode of interaction, like Text-based games and dialog systems. In this work, we aim to bridge this gap and unify these two approaches in a single environment for multimodal RL. We introduce an extension of the TextWorld cooking environment with the addition of visual clues interspersed throughout the environment. The goal is to force an RL agent to use both text and visual features to predict natural language action commands for solving the final task of cooking a meal. We enable variations and difficulties in our environment to emulate various interactive real-world scenarios. We present a baseline multimodal agent for solving such problems using CNN-based feature extraction from visual hints and LSTMs for textual feature extraction. We believe that our proposed visual-lingual environment will facilitate novel problem settings for the RL community.
BibTeX
@misc{carta2020visualhints,
  author = {Thomas Carta and Subhajit Chaudhury and Kartik Talamadupula and Michiaki Tatsubori},
  title = {{VisualHints: A Visual-Lingual Environment for Multimodal Reinforcement Learning}},
  year = {2020},
  eprint = {2010.13839},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.13839},
}
Figure