Large Language Models (LLMs) are brilliant poets and coders, but they are notoriously clumsy "embodied" agents. When asked to navigate a virtual kitchen to boil water or complete household chores in a simulator, they often lose the thread of long-term goals or fail to understand the specific "physics" of their environment.
A new paper, "Discriminator-Guided Embodied Planning for LLM Agent," introduces a framework called DGAP that addresses this by giving LLMs a "critic" that grades their actions one step at a time.
TL;DR
DGAP (Discriminator-Guided Action Optimization) uses a small set of expert demonstrations to train a discriminator that scores LLM actions at every step. By feeding these scores back into the prompt, the LLM learns to optimize its plan in real-time, significantly outperforming existing methods in complex, long-horizon tasks.
What Problem Does This Paper Address?
The core challenge in embodied AI is grounding. An LLM might understand the concept of boiling water, but it doesn't inherently know the specific, step-by-step action sequence required by a simulator like ScienceWorld.
Existing solutions usually fall into a few camps:
- In-context learning: The LLM tries to figure out the task from a few examples, but errors compound over long sequences. If it fails, it relies on outcome-supervised "reflection" at the very end, which is slow and expensive.
- Tree of Thought / Search: The model explores multiple paths, but this can be computationally heavy.
- Demonstration-based (Imitation Learning): Requires massive amounts of expert data to train a reliable policy.
The authors identify a "feedback gap": LLMs need dense, step-by-step signals to stay on track, but generating those signals usually requires prohibitive amounts of expert data.
Methods: Discriminator-Guided Action Optimization (DGAP)
To bridge this gap, the authors propose DGAP, which operates in three main phases:
- The Discriminator as a Scorer: Instead of training a massive policy from scratch, the researchers trained a lightweight RoBERTa model to act as a discriminator. Using a limited set of expert demonstrations and data augmentation (generating "random" bad actions and "augmented" mediocre ones), the discriminator learns to assign a score from 0 to 10 to any proposed action based on the current state and history.
- Process-Supervised Prompting (PSP): During planning, the LLM is shown a history of its previous actions and their corresponding scores. The prompt explicitly instructs the LLM to "maximize high-score actions cumulatively."
- Closed-Loop Refinement: If the LLM proposes an action and the discriminator gives it a low score (e.g., below 5), the agent doesn't execute it in the environment. Instead, it triggers a re-planning phase, forcing the LLM to try again until it finds a high-scoring action.
The Theory: The authors provide a proof showing that DGAP is mathematically equivalent to critic-regularized optimization in Reinforcement Learning. Essentially, the LLM acts as a sampler, and the discriminator acts as a "soft" filter that pushes the distribution toward optimal expert behavior.
Datasets and Key Insights
The framework was evaluated on two primary benchmarks:
- ScienceWorld: A complex text-based environment with 30 types of science experiments (e.g., testing conductivity, mixing paints).
- VirtualHome: A simulator for household chores. The authors also introduced Extreme-LONG, a more difficult version of VirtualHome created by concatenating multiple objectives into tasks requiring over 60 steps.
Why DGAP Works
- Numerical Sensitivity: LLMs are surprisingly good at "gradient descent in natural language." When shown that
Action Ascored a 4 andAction Bscored an 8, the model can effectively infer the "direction" of a betterAction C. - Catching Errors Early: By using the discriminator to block low-scoring actions before execution, the agent avoids the compounding error problem where one wrong turn at step 5 makes step 50 impossible.
- Small Data Efficiency: Training a discriminator is much easier than training a full policy. The system can generalize from just 10–30 expert trajectories per task.
Key Results
The researchers tested DGAP using GPT-4 and Llama3-70B as the planning backbones.
- Superior Performance: In ScienceWorld, DGAP-GPT4 achieved an overall score of 85.91, surpassing the previous state-of-the-art, SwiftSage (84.68).
- Long-Horizon Success: In the "NovelTasks" category of VirtualHome, DGAP-GPT4 reached a 72.17% success rate, nearly doubling the performance of "Inner Monologue" (46%) and significantly beating "Tree Planner" (52.33%).
- Efficiency: DGAP reached successful outcomes in fewer environment steps compared to baselines, as the discriminator-guided re-planning happens "in the head" of the agent rather than through costly trial-and-error in the actual environment.
Limitations and Future Work
While highly effective for complex tasks, DGAP has a few limitations:
- Short-Task Overhead: For very short, simple tasks, the discriminator doesn't provide a significant boost, as the LLM's internal reasoning is already sufficient.
- Data Preparation: The method still requires a "judicious use" of expert data and a specific data augmentation pipeline to train the RoBERTa scorer.
- Visual Grounding: When the authors tried to incorporate RGB images via Vision-Language Models (VLMs), performance dipped. Current prompts and discriminators aren't yet optimized to handle the "noise" of raw visual data compared to clean text descriptions.
Future work will likely focus on making the discriminator training more "plug-and-play" and improving the integration of visual sensors for real-world robotics.
Summary updated