Introduction: The Challenge of Collaborative Agency
Most research in natural language instruction following assumes a "passive" user providing commands in a static environment. However, real-world collaboration is inherently dynamic. For instance, if a human and a robot are cleaning a kitchen together, the human might complete a task originally assigned to the robot, or a robot's mistake might render a subsequent instruction nonsensical.
The authors identify two major gaps in current systems:
- Collaborative Agency: Agents must ground language in a world state that changes independently of their own actions due to external actors.
- Error Propagation: In sequential tasks, a single error often makes all following instructions invalid. Standard "all-or-nothing" metrics fail to capture an agent's ability to recover and remain useful after a mid-task hiccup.
Proposed Architecture: Two-Stage Plan and Action Generation
To address these challenges, the paper proposes a two-stage architecture that maps natural language to actions by separating high-level planning from low-level execution.
Stage 1: Plan Prediction
Using a modified LINGUNET architecture, the model maps the instruction and the current world state to four spatial probability distributions:
- Visitation: Where should the agent go?
- Goal: Which cards should be picked up?
- Avoid: Which cards should be ignored?
- No-pass: Which areas are obstacles?
Stage 2: Action Generation
A recurrent neural network (LSTM) takes these predicted "plans" (cropped to the agent's local view) and generates a sequence of discrete actions such as moving, turning, or picking up items.
Key Innovations for Robustness
- Recovery Training: To handle errors, the authors used a DAGGER-like approach, intentionally starting the agent in "incorrect" states (resulting from simulated prior mistakes) and training it to reach the intended goal from there.
- Implicit Action Discriminator: A component was added to help the model distinguish between executing an explicit instruction and performing "implicit" recovery actions, like moving back to a path after getting lost.
- Cascaded Evaluation: A new metric that evaluates an agent by starting it at every possible instruction point in a recorded game. This measures how much of the remaining task the agent can complete, providing a more granular view of performance.
Data Collection and Human Evaluation
The researchers collected a new dataset of 1,202 human-human games played via Amazon Mechanical Turk. The model was trained on 960 games and tested on approximately 120 games. Furthermore, the model was tested in live games with human "leaders" to see how real people adapt to the AI's behavior.
Analysis of Human-AI Interaction
- User Adaptation: In live evaluations, humans used significantly simpler language when leading the AI (8.5 tokens/instruction) compared to leading other humans (12.3 tokens), effectively "dumbing down" their speech to match the agent's perceived abilities.
- Redundancy as Robustness: Predicting multiple distributions (Goal, Avoid, etc.) instead of just a single trajectory made the model more resilient. Even if the predicted path was slightly off, the "Goal" distribution helped the agent stay focused on the right objects.
- The "Cascading" Problem: Without recovery training, 82% of instructions became impossible to follow because the agent was in the wrong starting position due to a previous error. This highlights why "single-instruction" benchmarks are insufficient for real-world robotics.
Experimental Results
The full model achieved a 17.9% "Proportion of Points Scored" on the test set using cascaded evaluation, vastly outperforming a standard Seq2Seq+Attention baseline (6.5%). In live games, human-AI pairs scored an average of 6.2 points, compared to 12.7 points for human-human pairs. While there is still a gap, it proves the agent is a functional collaborator. Ablation studies confirmed that the "Trajectory Distribution" is the most critical component for spatial planning.
Conclusion and Future Directions
Current limitations include the model's assumption of full observability during training and its "silence"—the follower cannot yet ask for clarification. Additionally, while the model handles positional errors well, it still struggles when it picks up the wrong card, as this changes the game logic. Future work will focus on partial observability, bidirectional communication, and more complex error recovery.
Summary updated