Discriminator-Guided Embodied Planning for LLM Agent

Haofu Qian, Chenjia Bai, Jiatao Zhang, Fei Wu, Wei Song, Xuelong LiICLR 2025 · OpenReview

Large Language Models (LLMs) are brilliant poets and coders, but they are notoriously clumsy "embodied" agents. When asked to navigate a virtual kitchen to boil water or complete household chores in a simulator, they often lose the thread of long-term goals or fail to understand the specific "physics" of their environment.

A new paper, "Discriminator-Guided Embodied Planning for LLM Agent," introduces a framework called DGAP that addresses this by giving LLMs a "critic" that grades their actions one step at a time.

TL;DR

DGAP (Discriminator-Guided Action Optimization) uses a small set of expert demonstrations to train a discriminator that scores LLM actions at every step. By feeding these scores back into the prompt, the LLM learns to optimize its plan in real-time, significantly outperforming existing methods in complex, long-horizon tasks.

What Problem Does This Paper Address?

The core challenge in embodied AI is grounding. An LLM might understand the concept of boiling water, but it doesn't inherently know the specific, step-by-step action sequence required by a simulator like ScienceWorld.

Comparison of planning frameworks

Existing solutions usually fall into a few camps:

  1. In-context learning: The LLM tries to figure out the task from a few examples, but errors compound over long sequences. If it fails, it relies on outcome-supervised "reflection" at the very end, which is slow and expensive.
  2. Tree of Thought / Search: The model explores multiple paths, but this can be computationally heavy.
  3. Demonstration-based (Imitation Learning): Requires massive amounts of expert data to train a reliable policy.

The authors identify a "feedback gap": LLMs need dense, step-by-step signals to stay on track, but generating those signals usually requires prohibitive amounts of expert data.

Methods: Discriminator-Guided Action Optimization (DGAP)

To bridge this gap, the authors propose DGAP, which operates in three main phases:

  1. The Discriminator as a Scorer: Instead of training a massive policy from scratch, the researchers trained a lightweight RoBERTa model to act as a discriminator. Using a limited set of expert demonstrations and data augmentation (generating "random" bad actions and "augmented" mediocre ones), the discriminator learns to assign a score from 0 to 10 to any proposed action based on the current state and history.
  2. Process-Supervised Prompting (PSP): During planning, the LLM is shown a history of its previous actions and their corresponding scores. The prompt explicitly instructs the LLM to "maximize high-score actions cumulatively."
  3. Closed-Loop Refinement: If the LLM proposes an action and the discriminator gives it a low score (e.g., below 5), the agent doesn't execute it in the environment. Instead, it triggers a re-planning phase, forcing the LLM to try again until it finds a high-scoring action.

The Theory: The authors provide a proof showing that DGAP is mathematically equivalent to critic-regularized optimization in Reinforcement Learning. Essentially, the LLM acts as a sampler, and the discriminator acts as a "soft" filter that pushes the distribution toward optimal expert behavior.

Datasets and Key Insights

The framework was evaluated on two primary benchmarks:

  • ScienceWorld: A complex text-based environment with 30 types of science experiments (e.g., testing conductivity, mixing paints).
  • VirtualHome: A simulator for household chores. The authors also introduced Extreme-LONG, a more difficult version of VirtualHome created by concatenating multiple objectives into tasks requiring over 60 steps.

Why DGAP Works

  • Numerical Sensitivity: LLMs are surprisingly good at "gradient descent in natural language." When shown that Action A scored a 4 and Action B scored an 8, the model can effectively infer the "direction" of a better Action C.
  • Catching Errors Early: By using the discriminator to block low-scoring actions before execution, the agent avoids the compounding error problem where one wrong turn at step 5 makes step 50 impossible.
  • Small Data Efficiency: Training a discriminator is much easier than training a full policy. The system can generalize from just 10–30 expert trajectories per task.

Key Results

The researchers tested DGAP using GPT-4 and Llama3-70B as the planning backbones.

  • Superior Performance: In ScienceWorld, DGAP-GPT4 achieved an overall score of 85.91, surpassing the previous state-of-the-art, SwiftSage (84.68).
  • Long-Horizon Success: In the "NovelTasks" category of VirtualHome, DGAP-GPT4 reached a 72.17% success rate, nearly doubling the performance of "Inner Monologue" (46%) and significantly beating "Tree Planner" (52.33%).
  • Efficiency: DGAP reached successful outcomes in fewer environment steps compared to baselines, as the discriminator-guided re-planning happens "in the head" of the agent rather than through costly trial-and-error in the actual environment.

Limitations and Future Work

While highly effective for complex tasks, DGAP has a few limitations:

  • Short-Task Overhead: For very short, simple tasks, the discriminator doesn't provide a significant boost, as the LLM's internal reasoning is already sufficient.
  • Data Preparation: The method still requires a "judicious use" of expert data and a specific data augmentation pipeline to train the RoBERTa scorer.
  • Visual Grounding: When the authors tried to incorporate RGB images via Vision-Language Models (VLMs), performance dipped. Current prompts and discriminators aren't yet optimized to handle the "noise" of raw visual data compared to clean text descriptions.

Future work will likely focus on making the discriminator training more "plug-and-play" and improving the integration of visual sensors for real-world robotics.

Summary updated

Abstract

Large Language Models (LLMs) have showcased remarkable reasoning capabilities in various domains, yet face challenges in complex embodied tasks due to the need for a coherent long-term policy and context-sensitive environmental understanding. Previous work performed LLM refinement relying on outcome-supervised feedback, which can be costly and ineffective. In this work, we introduce a novel framework, Discriminator-Guided Action Optimization (DGAP), for facilitating the optimization of LLM action plans via step-wise signals. Specifically, we employ a limited set of demonstrations to enable the discriminator to learn a score function, which assesses the alignment between LLM-generated actions and the underlying optimal ones at every step. Based on the discriminator, LLMs are prompted to generate actions that maximize the score, utilizing historical action-score pair trajectories as guidance. Under mild conditions, DGAP resembles critic-regularized optimization and has been demonstrated to achieve a stronger policy than the LLM planner. In experiments across different LLMs (GPT-4, Llama3-70B) in ScienceWorld and VirtualHome, our method achieves superior performance and better efficiency than previous methods.
BibTeX
@inproceedings{qian2025discriminator,
  author = {Haofu Qian and Chenjia Bai and Jiatao Zhang and Fei Wu and Wei Song and Xuelong Li},
  title = {{Discriminator-Guided Embodied Planning for LLM Agent}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=TjP1d8PP8l},
}
Figure