TL;DR
FILM (Following Instructions in Language with Modular methods) moves away from the "black-box" end-to-end models common in embodied AI. By using a modular architecture that combines semantic mapping, a smart search policy, and template-based language processing, it achieves state-of-the-art performance on the ALFRED benchmark—outperforming previous methods by over 8% while requiring significantly less training data.
What Problem Does This Paper Address?
The goal of Embodied Instruction Following (EIF) is to create agents that can execute complex, multi-step tasks in a 3D environment based on natural language commands (e.g., "Put a heated apple on the counter"). Unlike simple navigation, EIF requires the agent to understand compositional instructions, maintain a long-term memory of the environment, and interact with objects (picking, heating, slicing).
Most prior approaches relied on end-to-end imitation learning, where a neural network tries to map raw pixels and text directly to low-level actions. These "black-box" models often struggle with long-term planning, fail to generalize to new room layouts, and require massive amounts of "expert trajectory" data to learn basic spatial common sense.
Methods
The authors propose a modular system called FILM, which breaks the problem into four distinct, interpretable components:
- Language Processing (LP): Instead of processing text as a single vector, FILM uses two BERT-based sub-modules to translate high-level instructions into a sequence of discrete subtasks. It identifies the task type (e.g., "Clean & Place") and the specific objects involved (the "apple" and the "sink").
- Semantic Mapping: The agent builds a 2D top-down "semantic map" in real-time. It uses off-the-shelf depth estimation and instance segmentation (Mask R-CNN) to project egocentric 2D images into a 3D point cloud, which is then flattened into a map that tracks the locations of obstacles and various object categories.
- Semantic Search Policy: This is the "brain" of the exploration. Small objects (like a fork) are hard to see from across a room. FILM trains a Convolutional Neural Network (CNN) to predict the likely location of a goal object based on the layout of larger "receptacle" objects (e.g., "look for the fork near the table or sink").
- Deterministic Policy: Once a goal is identified (either a predicted search location or a spotted object), a pathfinder using the Fast Marching Method (FMM) calculates the most efficient route. A simple state machine then handles the interaction actions like "PickUp" or "Toggle."
Datasets
The model was evaluated on the ALFRED benchmark, a challenging dataset for embodied AI featuring 25,000+ language instructions for household tasks across diverse simulated rooms in AI2-THOR. The paper specifically focuses on the "Tests Unseen" split, where the agent is placed in entirely new room layouts it never encountered during training.
Key Insights and Ideas
- Modularity Beats End-to-End: By explicitly separating "where to look" (search policy) from "how to get there" (navigation), the agent becomes much more robust. It doesn't have to "re-learn" how to walk every time it learns a new word.
- Semantic Common Sense: The most innovative part of FILM is the Semantic Search Policy. It learns spatial correlations—knowing that a "pot" is likely on a "stove"—which allows it to explore efficiently rather than wandering randomly.
- Transparency: Because the map and the subtask list are explicit, researchers can see exactly why an agent failed (e.g., "it thought the apple was a potato" vs. "it couldn't find the microwave").
- Data Efficiency: FILM achieves SOTA results without using expert trajectories or low-level step-by-step instructions for training, proving that structured spatial representations are a powerful inductive bias.
Key Results
- New State-of-the-Art: FILM achieved a 24.46% success rate on the ALFRED "Tests Unseen" leaderboard. This was a massive 8.17% absolute improvement over the previous best model (HLSM) at the time of publication.
- Generalization: FILM's performance gap between "Seen" and "Unseen" environments is much smaller than end-to-end models, demonstrating superior generalization.
- Search Efficiency: The semantic search policy increased the success rate in large rooms significantly, as the agent spent less time "lost" in irrelevant areas.
Limitations and Future Work
- The "Closed Receptacle" Problem: FILM currently struggles when a goal object is hidden inside a closed cabinet or drawer. The agent doesn't yet have an "active search" strategy to open containers just to look inside.
- Perception Bottlenecks: Since FILM relies on pre-trained vision models (Mask R-CNN), any error in object detection or depth estimation propagates through the entire system.
- Template Rigidity: The language module currently maps instructions to fixed templates. The authors suggest that future work could use more flexible program synthesis to handle even more complex, non-templated instructions.
Summary updated