Abstract
Embodied dialogue instruction following requires an agent to complete a complex sequence of tasks from a natural language exchange. The recent introduction of benchmarks raises the question of how best to train and evaluate models for this multi-turn, multi-agent, long-horizon task. This paper contributes to that conversation, by arguing that imitation learning (IL) and related low-level metrics are actually misleading and do not align with the goals of embodied dialogue research and may hinder progress. We provide empirical comparisons of metrics, analysis of three models, and make suggestions for how the field might best progress. First, we observe that models trained with IL take spurious actions during evaluation. Second, we find that existing models fail to ground query utterances, which are essential for task completion. Third, we argue evaluation should focus on higher-level semantic goals. We will release code to additionally filter the data and benchmark models for improved evaluation.
Introduction
In the field of Embodied Dialogue, an agent must complete complex tasks (e.g., "Make me a coffee") by interacting with a human "commander" through natural language. While recent benchmarks like TEACh (Task-driven Embodied Agents that Chat) provide a platform for this research, a significant mismatch exists between current training objectives and ultimate research goals. Most models rely on Imitation Learning (IL), where the agent is trained to predict the exact next action a human took. However, human demonstrations are "ecologically valid"—they include mistakes, suboptimal paths, and "no-op" actions that do not contribute to the goal. This paper argues that forcing models to copy these low-level behaviors hinders their ability to generalize and understand semantic objectives.
Methodology
To investigate these challenges, we evaluate three distinct model architectures on the TEACh benchmark:
- Episodic Transformer (ET): An end-to-end model trained via behavior cloning that maps visual and text inputs directly to low-level actions.
- Symbiote: A modular system using a pretrained T5 model to predict high-level subgoals, which then guide a planner.
- FILM (Following Instructions with Modular Methods): A modular approach that builds a semantic map of the environment and uses a semantic policy to decide on high-level goals.
We contrast two primary evaluation tasks: Entire Dialogue History (EDH), which measures the ability to predict the next set of actions in a snippet (an imitation metric), and Trajectory from Dialogue (TfD), which requires the agent to complete the entire task from scratch based on the full dialogue (a goal-oriented metric).
The Pitfalls of Imitation Learning
Training agents to mimic human trajectories leads to the replication of spurious behaviors. In the TEACh dataset, human demonstrators often perform "No-Op" actions—useless interactions like picking up and immediately dropping an object. Models trained on IL, such as the Episodic Transformer, learn to replicate these errors nearly 50% of the time in unseen scenes.
Furthermore, the EDH metric can be misleading. Success rates on EDH are often inflated by deterministic sub-sequences (like the final "place object" action), allowing a model to appear competent while remaining unable to complete a full task (TfD). As shown in Figure 6, the misalignment between dialogue and demo actions across different sessions further complicates the learning process for end-to-end models.
Key Results and Discussion
The performance gap between imitation and actual task success is stark. On the TfD task in unseen environments, the baseline ET model achieved a success rate of only 0.5%. In contrast, the modular FILM model achieved 2.9%. While both are low, the modular approach is significantly more robust because it targets semantic states (e.g., "Object X is sliced") rather than mimicking specific movements. Notably, FILM exhibited 0% "No-Op" behaviors, whereas ET exhibited them in 48.1% of tasks.
Another critical failure point is Query Grounding. When an agent asks a question (e.g., "Where is the coffee?") and the commander provides the answer, current models largely ignore this information and continue searching randomly. Success rates for tasks requiring query grounding remain near zero across all tested architectures.
Conclusion
To advance embodied dialogue research, the community must shift away from simple behavior cloning. We recommend:
- Improving Query Grounding: Developing architectures that update internal world models based on dialogue information.
- Adopting Goal-Oriented Metrics: Shifting focus from trajectory copying to efficient state-reaching.
- Handling Noisy Data: Creating techniques to extract high-level intent from suboptimal human demonstrations without replicating the noise.
Summary updated