Abstract
Reinforcement Learning (RL) agents typically operate within single modalities, such as pixel-based visual inputs or text-based natural language. However, real-world tasks often require the simultaneous integration of diverse information sources. We introduce VisualHints, a novel multimodal RL environment that extends text-based games with visual floor plans. By requiring agents to "examine" visual hints to resolve textual ambiguities and avoid hazards, VisualHints provides a benchmark for spatial reasoning and multimodal fusion. We present a baseline architecture utilizing CNNs and LSTMs and evaluate its performance against state-of-the-art unimodal models, highlighting the unique challenges of cross-modal integration.
Introduction
Traditional RL environments often isolate modalities. Text-based games (TBGs) are effective for testing language understanding and long-term planning but suffer from partial observability, where the agent's knowledge is restricted to the current room's description. In contrast, human intelligence frequently relies on visual context—such as a map—to navigate complex environments and interpret verbal instructions.
VisualHints addresses this gap by modifying Microsoft’s TextWorld "Cooking Game" to include visual "hints." These hints are floor plans that the agent must utilize to navigate efficiently and avoid "Death Rooms"—high-penalty zones that lead to immediate failure. This setup forces the agent out of unimodal silos, requiring true multimodal synthesis to reach optimal solutions.
The VisualHints Environment
VisualHints is built as an extension layer over TextWorld, offering a customizable benchmark for researchers. The environment generates floor plans with varying difficulty levels and features that can be toggled to test specific agent capabilities:
- Death Rooms: Zones that impose heavy penalties, necessitating visual identification to avoid.
- Color Paths: Visual guides that can be used to simplify navigation.
- Room Importance: A feature that masks room names and replaces them with numerical priorities, testing the agent's ability to reason about abstract spatial data.
The environment includes evaluation sets for "Navigation Games" (ranging from 6 to 12 rooms) and "Non-Navigation Games" where the agent begins in the target kitchen.
Multimodal Agent Architecture
The proposed baseline agent employs an Advantage Actor-Critic (A2C) framework designed for multimodal fusion. The architecture consists of three primary encoding modules:
- Textual Module: Encodes game state descriptions and available natural language commands using an LSTM.
- Visual Module: Utilizes a CNN to extract features from floor plans. These features are processed by a bidirectional LSTM to capture spatial relationships and the global topology of the map.
- Fusion & Policy: Features from both modalities are concatenated. This fused representation is used by the policy network to select actions and by the value network to estimate the potential reward of the current state.
To ensure the visual module effectively interprets spatial data, it was pre-trained on various binary classification tasks, such as identifying room types or counting inaccessible rooms.
Experimental Results and Discussion
The multimodal baseline was compared against LeDeepChef, a state-of-the-art text-only model. In complex 12-room navigation tasks, the Visual-Lingual model achieved a success rate of 0.13 on the test set, compared to 0.31 for LeDeepChef.
While the multimodal baseline did not outperform the highly optimized text-only SOTA, these results establish a critical baseline for a more difficult problem class. Analysis of the agent's behavior provided several insights:
- Spatial Reasoning Bottleneck: Pre-training revealed that while agents easily identify room existence (0.99 F1), they struggle with topological reasoning, such as counting rooms blocked by hazards (0.8 F1). This indicates that "seeing" a map is distinct from "understanding" its spatial implications.
- Visual Attention: CNN filter analysis showed the model successfully learned to highlight specific rooms and identify the "death room" based on color and position.
Conclusion
VisualHints provides a necessary sandbox for developing agents capable of simultaneous visual and lingual reasoning. The primary challenge remains the effective fusion of disparate data types—pixels and words—within a massive natural language action space. Future work will explore more sophisticated fusion mechanisms, such as cross-modal attention, and evaluate the environment with advanced RL algorithms beyond A2C.
Summary updated