FILM: Following Instructions in Language with Modular Methods

So Yeon Min, Devendra Singh Chaplot, Pradeep Kumar Ravikumar, Yonatan Bisk, Ruslan Salakhutdinov

TL;DR

FILM (Following Instructions in Language with Modular methods) moves away from the "black-box" end-to-end models common in embodied AI. By using a modular architecture that combines semantic mapping, a smart search policy, and template-based language processing, it achieves state-of-the-art performance on the ALFRED benchmark—outperforming previous methods by over 8% while requiring significantly less training data.

Overview of the FILM agent executing a complex instruction.

What Problem Does This Paper Address?

The goal of Embodied Instruction Following (EIF) is to create agents that can execute complex, multi-step tasks in a 3D environment based on natural language commands (e.g., "Put a heated apple on the counter"). Unlike simple navigation, EIF requires the agent to understand compositional instructions, maintain a long-term memory of the environment, and interact with objects (picking, heating, slicing).

Most prior approaches relied on end-to-end imitation learning, where a neural network tries to map raw pixels and text directly to low-level actions. These "black-box" models often struggle with long-term planning, fail to generalize to new room layouts, and require massive amounts of "expert trajectory" data to learn basic spatial common sense.

Methods

The authors propose a modular system called FILM, which breaks the problem into four distinct, interpretable components:

The FILM modular architecture pipeline.
  1. Language Processing (LP): Instead of processing text as a single vector, FILM uses two BERT-based sub-modules to translate high-level instructions into a sequence of discrete subtasks. It identifies the task type (e.g., "Clean & Place") and the specific objects involved (the "apple" and the "sink").
  2. Semantic Mapping: The agent builds a 2D top-down "semantic map" in real-time. It uses off-the-shelf depth estimation and instance segmentation (Mask R-CNN) to project egocentric 2D images into a 3D point cloud, which is then flattened into a map that tracks the locations of obstacles and various object categories.
  3. Semantic Search Policy: This is the "brain" of the exploration. Small objects (like a fork) are hard to see from across a room. FILM trains a Convolutional Neural Network (CNN) to predict the likely location of a goal object based on the layout of larger "receptacle" objects (e.g., "look for the fork near the table or sink").
  4. Deterministic Policy: Once a goal is identified (either a predicted search location or a spotted object), a pathfinder using the Fast Marching Method (FMM) calculates the most efficient route. A simple state machine then handles the interaction actions like "PickUp" or "Toggle."

Datasets

The model was evaluated on the ALFRED benchmark, a challenging dataset for embodied AI featuring 25,000+ language instructions for household tasks across diverse simulated rooms in AI2-THOR. The paper specifically focuses on the "Tests Unseen" split, where the agent is placed in entirely new room layouts it never encountered during training.

Key Insights and Ideas

  • Modularity Beats End-to-End: By explicitly separating "where to look" (search policy) from "how to get there" (navigation), the agent becomes much more robust. It doesn't have to "re-learn" how to walk every time it learns a new word.
  • Semantic Common Sense: The most innovative part of FILM is the Semantic Search Policy. It learns spatial correlations—knowing that a "pot" is likely on a "stove"—which allows it to explore efficiently rather than wandering randomly.
  • Transparency: Because the map and the subtask list are explicit, researchers can see exactly why an agent failed (e.g., "it thought the apple was a potato" vs. "it couldn't find the microwave").
  • Data Efficiency: FILM achieves SOTA results without using expert trajectories or low-level step-by-step instructions for training, proving that structured spatial representations are a powerful inductive bias.

Key Results

  • New State-of-the-Art: FILM achieved a 24.46% success rate on the ALFRED "Tests Unseen" leaderboard. This was a massive 8.17% absolute improvement over the previous best model (HLSM) at the time of publication.
  • Generalization: FILM's performance gap between "Seen" and "Unseen" environments is much smaller than end-to-end models, demonstrating superior generalization.
  • Search Efficiency: The semantic search policy increased the success rate in large rooms significantly, as the agent spent less time "lost" in irrelevant areas.

Limitations and Future Work

  • The "Closed Receptacle" Problem: FILM currently struggles when a goal object is hidden inside a closed cabinet or drawer. The agent doesn't yet have an "active search" strategy to open containers just to look inside.
  • Perception Bottlenecks: Since FILM relies on pre-trained vision models (Mask R-CNN), any error in object detection or depth estimation propagates through the entire system.
  • Template Rigidity: The language module currently maps instructions to fixed templates. The authors suggest that future work could use more flexible program synthesis to handle even more complex, non-templated instructions.

Summary updated

Abstract

Recent methods for embodied instruction following are typically trained end-to-end using imitation learning. This often requires the use of expert trajectories and low-level language instructions. Such approaches assume that neural states will integrate multimodal semantics to perform state tracking, building spatial memory, exploration, and long-term planning. In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal. Our modular method achieves SOTA performance (24.46 %) with a substantial (8.17 % absolute) gap from previous work while using less data by eschewing both expert trajectories and low-level instructions. Leveraging low-level language, however, can further increase our performance (26.49 %). Our findings suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance, even in the absence of expert trajectories or low-level instructions.
BibTeX
@inproceedings{min2022film,
  author = {So Yeon Min and Devendra Singh Chaplot and Pradeep Kumar Ravikumar and Yonatan Bisk and Ruslan Salakhutdinov},
  title = {{FILM: Following Instructions in Language with Modular Methods}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=qI4542Y2s1D},
}
Figure