Executing Instructions in Situated Collaborative Interactions

Alane Suhr, Claudia Yan, Jacob Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, Yoav Artzi

Abstract

We study a collaborative scenario where a user not only instructs a system to complete tasks, but also acts alongside it. This allows the user to adapt to the system abilities by changing their language or deciding to simply accomplish some tasks themselves, and requires the system to effectively recover from errors as the user strategically assigns it new goals. We build a game environment to study this scenario, and learn to map user instructions to system actions. We introduce a learning approach focused on recovery from cascading errors between instructions, and modeling methods to explicitly reason about instructions with multiple goals. We evaluate with a new evaluation protocol using recorded interactions and online games with human users, and observe how users adapt to the system abilities.

Introduction

Most instruction-following research assumes a static world where only the AI agent acts. However, real-world collaboration is a two-way street: a human might give an instruction and then immediately start performing a related task themselves. This introduces two major challenges: dynamic environments where the world state changes due to user actions, and error propagation, where failing one part of a multi-step instruction makes subsequent steps impossible.

To address these, researchers at Cornell, IBM, and Intel introduced CEREALBAR, a new environment and dataset for studying how AI agents follow natural language instructions in collaborative settings where both the human and the agent act simultaneously.

The CEREALBAR Environment

The CEREALBAR game is a 3D environment where two players, a "Leader" and a "Follower," collaborate to collect sets of cards. The Leader has a global view of the map and provides instructions, while the Follower has a limited first-person view and executes actions.

The CEREALBAR environment showing Leader and Follower views with sample instructions.

The dataset consists of 1,202 human-human games collected via Amazon Mechanical Turk, containing nearly 24,000 individual instructions. This rich dataset captures how humans naturally communicate and coordinate in a shared space.

Model Architecture

The proposed "Follower" agent uses a two-stage architecture designed for interpretability and robustness.

The two-stage model architecture: Plan Prediction (Stage 1) and Action Generation (Stage 2).

Stage 1: Plan Prediction

Using a modified LINGUNET (a language-conditioned image-to-image network), the model takes the leader's text and a top-down view of the world to predict four spatial probability distributions:

  • Trajectory: Where the agent should visit.
  • Goals: Which items to pick up.
  • Avoidance: Which items to stay away from.
  • Impassable: Physical obstacles.

Stage 2: Action Generation

A recurrent neural network (LSTM) then "reads" these predicted maps to generate a sequence of discrete actions (move forward, turn, etc.). By forcing the model to predict a "goal map" before generating actions, the system becomes much more interpretable; researchers can visualize where the AI intends to go.

Learning to Recover

To handle cascading errors, the authors used a data augmentation technique similar to DAGGER. They intentionally started the agent in "incorrect" positions (simulating a previous mistake) and used a heuristic oracle to teach it how to get back on track. A discriminator was also added to help the model distinguish between executing a direct command and performing "implicit" recovery actions.

Evaluation and Results

The authors introduced a Cascaded Evaluation protocol. Instead of just testing if an agent can finish a whole game from the start, they test the agent starting from every single instruction point in a recorded game. This measures how well the agent performs on "the rest of the task" regardless of early-game failures.

Key Results

  • Performance: The full model significantly outperformed a standard Sequence-to-Sequence (Seq2Seq) baseline. On the cascaded evaluation, the model followed 15.4% of instructions and earned 17.9% of potential points, compared to 8.7% and 6.5% for the baseline.
  • Human Evaluation: In live games with humans, the agent scored an average of 6.2 points per game. While lower than human-human pairs (12.7), the agent was a functional partner.
  • Human Adaptation: In human-agent trials, users adapted to the AI's weaknesses by using simpler language and taking over difficult tasks themselves.

Conclusion

While the model shows promise, it currently assumes full observability of the map, whereas the actual game Follower has a limited field of view. Future work includes moving to partial observability and enabling bidirectional communication so the Follower can ask for clarification. Correcting selection errors (e.g., picking up the wrong card) also remains a challenge requiring more complex reasoning about game rules.

Summary updated

Abstract

We study a collaborative scenario where a user not only instructs a system to complete tasks, but also acts alongside it. This allows the user to adapt to the system abilities by changing their language or deciding to simply accomplish some tasks themselves, and requires the system to effectively recover from errors as the user strategically assigns it new goals. We build a game environment to study this scenario, and learn to map user instructions to system actions. We introduce a learning approach focused on recovery from cascading errors between instructions, and modeling methods to explicitly reason about instructions with multiple goals. We evaluate with a new evaluation protocol using recorded interactions and online games with human users, and observe how users adapt to the system abilities.
BibTeX
@inproceedings{suhr2019executing,
  author = {Alane Suhr and Claudia Yan and Jacob Schluger and Stanley Yu and Hadi Khader and Marwa Mouallem and Iris Zhang and Yoav Artzi},
  title = {{Executing Instructions in Situated Collaborative Interactions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1218},
}
Figure