TL;DR
Bilevel-LLM introduces a leader-follower reinforcement learning framework that automatically learns what questions to ask to make downstream decision-making easier. By training a prompt-generation policy to minimize the action policy’s uncertainty (entropy) while using an LLM for Chain-of-Thought (CoT) reasoning, the system guides low-level actions without relying on brittle, handcrafted prompt engineering.
What Problem Does This Paper Address?
While Large Language Models (LLMs) equipped with Chain-of-Thought (CoT) reasoning possess immense world knowledge, using them for sequential decision-making in reinforcement learning (RL) faces two major hurdles:
- Prompt Brittleness: Standard CoT relies heavily on human-crafted prompts. Manually tuning prompts for dynamic, state-dependent environments requires substantial labor and often fails to generalize.
- Action Grounding Gap: LLM-generated thoughts are written in natural language for human comprehension, making them frequently ambiguous or hard to interpret for low-level control policies.
The authors ask: Can an agent learn to autonomously generate state-appropriate prompts that induce helpful, decisive CoT reasoning for downstream action execution?
Methods
The authors structure the problem as a leader-follower bilevel optimization problem divided into three connected components:
[ Observations ] ──> [ Prompt Policy (π_ϕ) ] ──(Question)──> [ Frozen LLM (π_re) ]
│ (CoT Thoughts)
▼
[ Next State ] <──(Action)── [ Action Policy (π_θ) ] <────────────────┘
- Prompt-Generation Policy ($\pi_\phi$): Observes recent environment history and selects a relevant prompt question. Rather than using environmental reward, it is trained via policy gradient (REINFORCE) to minimize the entropy (uncertainty) of the action policy.
- Thought Reasoning Policy ($\pi_{re}$): A frozen LLM (GPT-3.5-Turbo) that receives the selected question and a description of the current situation to produce step-by-step CoT reasoning.
- Action Policy ($\pi_\theta$): A lightweight pre-trained language model (Flan-T5 Small, <1B parameters) trained via Proximal Policy Optimization (PPO). It takes both the raw environment observation and the CoT thought, learning to ground language reasoning into discrete environment actions while maximizing environmental reward.
To reduce API calls, the authors abstract environmental states into shared "situations" (e.g., slicing lettuce involves the same core sub-steps regardless of the agent's exact grid coordinates) and cache CoT responses.
Datasets & Environments
The framework is evaluated across three interactive decision-making benchmarks:
- ChainWorld: A 1D navigation chain tested in two variants: Full (complete observation of agent position and high-reward goal) and Partial (partial position observation, requiring reasoning over history).
- FourRoom: A navigation benchmark where an agent must traverse four interconnected rooms and hallways to reach a randomly placed target.
- Overcooked: A multi-step coordination and cooking gridworld requiring agents to prepare recipes (e.g., chopping tomatoes, assembling tomato-lettuce salads) across standard ($5 \times 4$) and large ($7 \times 7$) kitchen layouts.
Key Insights and Ideas
- Entropy as an Intrinsic Reward for Prompts: Rather than optimizing the prompt generator on sparse environment rewards, using negative action-policy entropy forces the prompt generator to ask questions whose CoT answers make the action policy confident about its next move.
- Bilevel Decoupling: Decoupling high-level reasoning (frozen GPT-3.5) from low-level execution (fine-tuned Flan-T5) avoids the costly fine-tuning of massive LLMs while eliminating the need for rigid, rule-based parsing modules.
- Zero-Shot LLMs Fail at Grounding: Directly prompting GPT-3.5 (even with CoT) performs poorly in complex environments due to world-model drift and an inability to ground abstract multi-step plans into valid, sequential environment actions.
Key Results
- Baseline Outperformance: Bilevel-LLM consistently outperforms Vanilla PPO, GFlan (Flan-T5 tuned without prompts), zero-shot GPT-3.5, and zero-shot GPT-3.5 with CoT across all benchmarks.
- Kitchen Efficiency: On the Overcooked (Salad) task, Bilevel-LLM achieves a normalized area-under-the-curve (AUC) reward near 1.0, whereas GFlan peaks around 0.9 with significantly higher variance.
- Ablations on Prompting:
- Replacing the policy-gradient prompt policy with random selection or Upper Confidence Bound (UCB) bandits led to steep performance degradation, proving the importance of conditioning prompt selection on state history.
- Training the prompt policy on environmental return rather than action-policy entropy resulted in slower convergence and higher policy entropy.
- Fully Automated Prompts (Bilevel-LLM-Auto): When candidate prompts were auto-generated by GPT-3.5 from task descriptions rather than written by humans, Bilevel-LLM maintained competitive performance matching human-crafted prompt sets.
Limitations and Future Work
- Discrete Prompt Candidates: The prompt generator currently selects from a candidate pool (either human-authored or pre-generated via GPT-3.5) by projecting embeddings, rather than generating free-form prompt tokens fully end-to-end.
- API Latency and Caching: The pipeline depends on external API calls for CoT generation. Although state-to-situation abstraction mitigates this, scaling to non-discrete or continuous environments where state abstraction is non-trivial remains an open challenge.
- Benchmark Scope: Experiments are limited to discrete gridworlds and navigation domains; validating the framework on complex 3D embodied agents (e.g., Habitat, ALFRED) is left for future exploration.
Summary updated