EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL

Thomas Carta, Sylvain Lamprier, Pierre-Yves Oudeyer, Olivier Sigaud

Abstract

Reinforcement Learning (RL) agents often struggle with sparse rewards in complex, language-conditioned tasks. Traditional reward shaping requires manual expert intervention to define sub-goals. This paper introduces EAGER (Exploit question-Answering Grounding for effective Exploration in RL), a framework that automates reward shaping by leveraging the compositional nature of language. By generating auxiliary "quizzes" through Question Generation (QG) and evaluating agent trajectories via Question Answering (QA), EAGER provides a dense internal reward signal. Experimental results on the BabyAI platform demonstrate that EAGER significantly improves sample efficiency and outperforms existing state-of-the-art methods without requiring human-defined sub-goals.

Introduction

In language-conditioned reinforcement learning (LC-RL), agents are tasked with executing complex instructions such as "Pick up the red key, open the door, and put the ball next to the box." A primary bottleneck in training these agents is the "sparse reward" problem: the agent typically only receives feedback upon total task completion. In environments with long-horizon dependencies, the probability of an agent stumbling upon the correct sequence of actions by chance is infinitesimally low.

While reward shaping—adding intermediate rewards to guide the agent—is a proven solution, it traditionally relies on human experts to manually decompose instructions into actionable sub-goals. EAGER addresses this limitation by asking: Can an agent leverage the inherent structure of language to autonomously generate its own intermediate rewards?

The EAGER Framework

EAGER introduces an automated loop for generating intrinsic rewards based on the agent's interaction with the environment. The framework consists of four primary components:

1. Question Generation (QG)

When an agent receives a goal (e.g., "Put the red ball next to the blue box"), the QG module automatically generates auxiliary objectives. It achieves this by masking nouns and adjectives within the instruction, creating questions like "Put the [MASK] ball next to the blue box." These masked entities represent the essential attributes and objects the agent must interact with to succeed.

2. Question Answering (QA)

A pre-trained QA module, implemented as an episodic transformer, monitors the agent’s trajectory. At each timestep, the QA module attempts to answer the generated questions based on the agent's visual observations and actions. To prevent the agent from receiving rewards for lucky guesses, the QA module includes a "no_answer" token; it only provides a specific answer when the trajectory contains sufficient evidence to identify the masked word with high confidence.

3. Intrinsic Reward Mechanism

The agent receives an intrinsic reward whenever the QA module correctly identifies a masked word. This mechanism incentivizes the agent to interact with relevant objects in a way that "explains" the goal through its actions. By successfully "answering" the self-generated quiz, the agent confirms it has encountered the necessary components of the task.

4. Reward Neutralization

A common risk in reward shaping is "policy drift," where an agent learns to "farm" intrinsic rewards while ignoring the actual goal. EAGER mitigates this by subtracting the total accumulated intrinsic reward from the final extrinsic reward. This ensures policy invariance, meaning the optimal path to the final goal remains consistent while the exploration process becomes significantly more directed.

Experimental Evaluation

The authors evaluated EAGER using the BabyAI platform, a MiniGrid-based environment designed for grounded language learning. The QA module was pre-trained on a dataset of 7,500 noisy, bot-generated trajectories per task. Testing spanned several levels of complexity, including PutNextTo, Unlock, and Sequence tasks (which require chaining multiple instructions).

Key Results

  • Superiority over SOTA: EAGER outperformed ELLA, the previous state-of-the-art for language reward shaping. Notably, while ELLA requires expert intervention to define sub-goal categories, EAGER is fully automated.
  • Sample Efficiency: In the "Unlock-Medium" task, EAGER reached high success rates significantly faster than standard PPO and RIDE (a popular non-language intrinsic motivation baseline).
  • Handling Temporal Complexity: In "Sequence" tasks involving temporal constraints (e.g., "do X before Y"), EAGER maintained strong performance, effectively decomposing the goals where other methods struggled.

Discussion and Insights

EAGER demonstrates that language is a natural tool for task decomposition. By masking words, the system identifies the essential entities the agent must ground in the environment. Qualitative analysis revealed that EAGER agents produce "cleaner" and more spatially compact trajectories; because they are rewarded for unambiguously explaining the goal through action, they wander less and exhibit fewer redundant movements than baseline agents.

Furthermore, the framework proved robust to QA inaccuracies. The authors found that as long as the QA success rate remained above approximately 52%, the agent still derived significant benefits from the shaped rewards, suggesting the framework does not require a perfect oracle to be effective.

Limitations and Future Work

Despite its success, EAGER currently relies on offline QA training using pre-existing trajectories. Future research could explore "autotelic" strategies where the QA module is learned online alongside the agent. Additionally, while BabyAI uses synthetic language, applying EAGER to more diverse, natural language benchmarks like ALFRED remains a critical next step. Finally, moving beyond simple word masking to more complex, multi-hop question generation could allow the framework to guide agents through even more intricate, multi-stage tasks.

Conclusion

EAGER represents a significant step toward fully autonomous language-conditioned RL. By turning instructions into a self-generated quiz, the framework allows agents to bridge the gap between sparse extrinsic rewards and the complex requirements of linguistic goals, achieving higher efficiency without the need for human-engineered sub-goals.

Summary updated

BibTeX
@inproceedings{carta2022eager,
  author = {Thomas Carta and Sylvain Lamprier and Pierre-Yves Oudeyer and Olivier Sigaud},
  title = {{EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=F9I5uYmpzwA},
}
Figure