Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents
Introduction
For an autonomous agent to follow a command like "Navigate to the object in front of the monster," it must perform language grounding: the ability to map abstract words ("front," "monster") to raw pixels. While deep reinforcement learning (RL) has made strides in training agents from scratch, existing methods for fusing vision and language often fall short. Simple techniques like concatenating vectors or "gating" (multiplying visual features by a language-based scalar) are often too rigid for complex spatial reasoning in partially observable environments.
Methodology: Guided Feature Transformation (GFT)
The core innovation presented is the Guided Feature Transformation (GFT) module. Unlike standard methods like Feature-wise Linear Modulation (FiLM), which only allow language to scale or shift visual feature maps, GFT models the interaction as a series of matrix transformations.
Mathematically, GFT acts as a "generalized gated network." While FiLM only scales features in their original space, GFT can rotate the axes of the visual feature space, scale them, and rotate them again based on the specific command. This "Rotate-Scale-Rotate" mechanism allows the agent to redefine the coordinate system of the features, isolating relevant information for specific spatial tasks. The module is embedded within a larger navigation agent using a Convolutional Neural Network (CNN) for vision and a Gated Recurrent Unit (GRU) for history, trained end-to-end using an Advantage Actor-Critic (A2C) algorithm.
Experimental Environments
The authors evaluate the GFT module in two distinct environments designed to test spatial reasoning and language grounding.
The first, XWORLD2D, is a grid-world with partial observability where the agent sees only a 5x5 area. The second, XWORLD3D, is a newly introduced environment featuring perspective distortion, various object scales, and random orientations, mimicking real-world visual challenges more closely.
Tasks across these environments include standard navigation (nav), moving near objects (nav_nr), moving between objects (nav_bw), avoiding specific objects (nav_avoid), and following relative directions (nav_dir).
Results and Discussion
GFT-2 (utilizing two transformation steps) consistently outperformed state-of-the-art modules like FiLM and Gated networks. In the challenging 3D nav_dir (spatial direction) task, GFT-2 achieved a 61.4% success rate, significantly higher than FiLM (~52%) and Gated networks (~48%). In 2D navigation, GFT-2 reached 70.7% success compared to FiLM’s 58.8%.
Key insights from the study include:
- Expressivity through Rotation: Redefining the feature coordinate system through rotation is crucial for isolating relevant information in spatial tasks.
- Depth Matters: Multi-stage reasoning (J=2) provides a significant performance jump over a single step, suggesting that sequential transformations help resolve complex linguistic constraints.
- Generalization: GFT agents maintained their lead on larger maps than those used in training, suggesting they learned general grounding principles rather than just memorizing maps.
Conclusion
GFT represents a powerful, portable module for language grounding. While it carries a higher computational cost due to the generation of large projection matrices, its superior performance in complex 3D navigation tasks highlights the value of expressive, differentiable feature transformations in embodied AI.
Summary updated