FILM: Following Instructions in Language with Modular Methods

So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, Ruslan Salakhutdinov

Abstract

Recent methods for embodied instruction following are typically trained end-to-end using imitation learning. This often requires the use of expert trajectories and low-level language instructions. Such approaches assume that neural states will integrate multimodal semantics to perform state tracking, building spatial memory, exploration, and long-term planning. In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal. Our modular method achieves SOTA performance (24.46 %) with a substantial (8.17 % absolute) gap from previous work while using less data by eschewing both expert trajectories and low-level instructions. Leveraging low-level language, however, can further increase our performance (26.49 %). Our findings suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance, even in the absence of expert trajectories or low-level instructions.

Introduction

Embodied Instruction Following (EIF) is a challenging domain in AI that requires agents to execute multi-step tasks in 3D environments based on natural language commands, such as "Put a heated apple on the counter." Traditional approaches to EIF often rely on end-to-end imitation learning, attempting to map raw pixels and text directly to actions. These "black box" models frequently struggle with long-horizon tasks, fail to maintain reliable spatial memory, and require massive amounts of data, including step-by-step low-level instructions and expert trajectories.

Overview of the FILM framework showing instruction decomposition and subtask execution.

FILM (Following Instructions in Language with Modular Methods) addresses these limitations by departing from end-to-end trends. It introduces a modular design that utilizes explicit spatial memory and structured state tracking, allowing for more efficient and interpretable task execution.

Methodology

FILM decomposes the EIF problem into four distinct, specialized modules that work in concert to achieve the high-level goal:

  1. Language Processing (LP): Utilizing BERT-based models, this module translates high-level instructions into a structured sequence of subtasks. It identifies the instruction type (e.g., "Clean & Place") and extracts relevant arguments like the target object and receptacle.
  2. Semantic Mapping: As the agent explores, it projects egocentric visual data (processed via Mask R-CNN for segmentation and depth estimation) into a top-down 2D semantic map. This map serves as an explicit spatial memory, tracking observed object categories and explored areas.
  3. Semantic Search Policy: This core innovation addresses the difficulty of finding small objects. A CNN is trained to predict the likely location of goal objects based on the spatial distribution of larger "receptacle" objects (like sinks or fridges) already recorded on the map.
  4. Deterministic Policy: Once a target or search goal is identified, the agent uses the Fast Marching Method for path planning and a state machine to execute specific interactions (e.g., "Pick Up," "Toggle On").
The modular architecture of FILM, integrating language processing, semantic mapping, and search policies.

Evaluation and Results

The authors evaluated FILM on the ALFRED benchmark, which features over 25,000 language directives across 120 indoor rooms. FILM set a new state-of-the-art record at the time of publication, achieving a 24.46% success rate on "Tests Unseen" (new layouts). This represents a significant 8.17% absolute increase over previous models like HLSM.

Crucially, FILM demonstrated superior generalization, maintaining consistent performance across both seen and unseen environments. It achieved these results with higher data efficiency, as it did not require expert trajectories or low-level step-by-step instructions to succeed.

Conclusion and Future Work

The success of FILM highlights that explicit spatial memory and semantic priors for exploration are more effective for long-term tasks than implicit neural states. The modularity of the system also provides high interpretability, allowing researchers to identify specific failure points in perception or language processing.

Future work aims to address current limitations, such as the inability to autonomously search inside closed receptacles (like drawers) without explicit instruction. Additionally, while the template-based language processing is robust, moving toward more flexible program synthesis could further improve the agent's ability to handle diverse human phrasing.

Summary updated

Abstract

Recent methods for embodied instruction following are typically trained end-to-end using imitation learning. This often requires the use of expert trajectories and low-level language instructions. Such approaches assume that neural states will integrate multimodal semantics to perform state tracking, building spatial memory, exploration, and long-term planning. In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal. Our modular method achieves SOTA performance (24.46 %) with a substantial (8.17 % absolute) gap from previous work while using less data by eschewing both expert trajectories and low-level instructions. Leveraging low-level language, however, can further increase our performance (26.49 %). Our findings suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance, even in the absence of expert trajectories or low-level instructions.
BibTeX
@inproceedings{min2022film,
  author = {So Yeon Min and Devendra Singh Chaplot and Pradeep Ravikumar and Yonatan Bisk and Ruslan Salakhutdinov},
  title = {{FILM: Following Instructions in Language with Modular Methods}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=qI4542Y2s1D},
}
Figure