Ask more, know better: Reinforce-Learned Prompt Questions for Decision Making with Large Language Models

Xue Yan, Yan Song, Xinyu Cui, Filippos Christianos, Haifeng Zhang, David Henry Mguni, Jun WangSubmission to ICLR 2024 · Rejected

TL;DR

Bilevel-LLM introduces a leader-follower reinforcement learning framework that automatically learns what questions to ask to make downstream decision-making easier. By training a prompt-generation policy to minimize the action policy’s uncertainty (entropy) while using an LLM for Chain-of-Thought (CoT) reasoning, the system guides low-level actions without relying on brittle, handcrafted prompt engineering.

What Problem Does This Paper Address?

While Large Language Models (LLMs) equipped with Chain-of-Thought (CoT) reasoning possess immense world knowledge, using them for sequential decision-making in reinforcement learning (RL) faces two major hurdles:

  1. Prompt Brittleness: Standard CoT relies heavily on human-crafted prompts. Manually tuning prompts for dynamic, state-dependent environments requires substantial labor and often fails to generalize.
  2. Action Grounding Gap: LLM-generated thoughts are written in natural language for human comprehension, making them frequently ambiguous or hard to interpret for low-level control policies.

The authors ask: Can an agent learn to autonomously generate state-appropriate prompts that induce helpful, decisive CoT reasoning for downstream action execution?

Methods

The authors structure the problem as a leader-follower bilevel optimization problem divided into three connected components:

[ Observations ] ──> [ Prompt Policy (π_ϕ) ] ──(Question)──> [ Frozen LLM (π_re) ]
                                                                       │ (CoT Thoughts)
                                                                       ▼
[ Next State ] <──(Action)── [ Action Policy (π_θ) ] <────────────────┘
  1. Prompt-Generation Policy ($\pi_\phi$): Observes recent environment history and selects a relevant prompt question. Rather than using environmental reward, it is trained via policy gradient (REINFORCE) to minimize the entropy (uncertainty) of the action policy.
  2. Thought Reasoning Policy ($\pi_{re}$): A frozen LLM (GPT-3.5-Turbo) that receives the selected question and a description of the current situation to produce step-by-step CoT reasoning.
  3. Action Policy ($\pi_\theta$): A lightweight pre-trained language model (Flan-T5 Small, <1B parameters) trained via Proximal Policy Optimization (PPO). It takes both the raw environment observation and the CoT thought, learning to ground language reasoning into discrete environment actions while maximizing environmental reward.

To reduce API calls, the authors abstract environmental states into shared "situations" (e.g., slicing lettuce involves the same core sub-steps regardless of the agent's exact grid coordinates) and cache CoT responses.

Datasets & Environments

The framework is evaluated across three interactive decision-making benchmarks:

  • ChainWorld: A 1D navigation chain tested in two variants: Full (complete observation of agent position and high-reward goal) and Partial (partial position observation, requiring reasoning over history).
  • FourRoom: A navigation benchmark where an agent must traverse four interconnected rooms and hallways to reach a randomly placed target.
  • Overcooked: A multi-step coordination and cooking gridworld requiring agents to prepare recipes (e.g., chopping tomatoes, assembling tomato-lettuce salads) across standard ($5 \times 4$) and large ($7 \times 7$) kitchen layouts.

Key Insights and Ideas

  • Entropy as an Intrinsic Reward for Prompts: Rather than optimizing the prompt generator on sparse environment rewards, using negative action-policy entropy forces the prompt generator to ask questions whose CoT answers make the action policy confident about its next move.
  • Bilevel Decoupling: Decoupling high-level reasoning (frozen GPT-3.5) from low-level execution (fine-tuned Flan-T5) avoids the costly fine-tuning of massive LLMs while eliminating the need for rigid, rule-based parsing modules.
  • Zero-Shot LLMs Fail at Grounding: Directly prompting GPT-3.5 (even with CoT) performs poorly in complex environments due to world-model drift and an inability to ground abstract multi-step plans into valid, sequential environment actions.

Key Results

  • Baseline Outperformance: Bilevel-LLM consistently outperforms Vanilla PPO, GFlan (Flan-T5 tuned without prompts), zero-shot GPT-3.5, and zero-shot GPT-3.5 with CoT across all benchmarks.
  • Kitchen Efficiency: On the Overcooked (Salad) task, Bilevel-LLM achieves a normalized area-under-the-curve (AUC) reward near 1.0, whereas GFlan peaks around 0.9 with significantly higher variance.
  • Ablations on Prompting:
  • Replacing the policy-gradient prompt policy with random selection or Upper Confidence Bound (UCB) bandits led to steep performance degradation, proving the importance of conditioning prompt selection on state history.
  • Training the prompt policy on environmental return rather than action-policy entropy resulted in slower convergence and higher policy entropy.
  • Fully Automated Prompts (Bilevel-LLM-Auto): When candidate prompts were auto-generated by GPT-3.5 from task descriptions rather than written by humans, Bilevel-LLM maintained competitive performance matching human-crafted prompt sets.

Limitations and Future Work

  • Discrete Prompt Candidates: The prompt generator currently selects from a candidate pool (either human-authored or pre-generated via GPT-3.5) by projecting embeddings, rather than generating free-form prompt tokens fully end-to-end.
  • API Latency and Caching: The pipeline depends on external API calls for CoT generation. Although state-to-situation abstraction mitigates this, scaling to non-discrete or continuous environments where state abstraction is non-trivial remains an open challenge.
  • Benchmark Scope: Experiments are limited to discrete gridworlds and navigation domains; validating the framework on complex 3D embodied agents (e.g., Habitat, ALFRED) is left for future exploration.

Summary updated

Abstract

Large language models (LLMs) demonstrate their promise in tackling complicated practical challenges by combining action-based policies with chain of thought (CoT) reasoning. Having high-quality prompts on hand, however, is vital to the framework’s effectiveness. Currently, these prompts are handcrafted utilising extensive human labor, resulting in CoT policies that frequently fail to generalise. Human intervention is also required in order to develop grounding functions that ensure low-level controllers appropriately process CoT reasoning. In this paper, we take the first step towards a fully integrated end-to-end framework for task-solving in real settings employing complicated reasoning. To that purpose, we offer a new leader-follower bilevel framework capable of learning to ask relevant questions (prompts) and subsequently undertaking reasoning to guide the learning of actions to be performed in an environment. A good prompt should make introspective revisions based on historical findings, leading the CoT to consider the anticipated goals. A prompt-generator policy has its own aim in our system, allowing it to adapt to the action policy and automatically root the CoT process towards outputs that lead to decisive, high-performing actions. Meanwhile, the action policy is learning how to use the CoT outputs to take specific actions. Our empirical data reveal that our system outperforms leading methods in agent learning benchmarks such as Overcooked and FourRoom.
BibTeX
@misc{yan2023ask,
  author = {Xue Yan and Yan Song and Xinyu Cui and Filippos Christianos and Haifeng Zhang and David Henry Mguni and Jun Wang},
  title = {{Ask more, know better: Reinforce-Learned Prompt Questions for Decision Making with Large Language Models}},
  year = {2023},
  eprint = {2310.18127},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.18127},
}
Figure