Using Natural Language for Reward Shaping in Reinforcement Learning

Prasoon Goyal, Scott Niekum, Raymond J. Mooney

Abstract

Recent reinforcement learning (RL) approaches have shown strong performance in complex domains such as Atari games, but are often highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the agent intermediate rewards for progress towards the goal. However, designing appropriate shaping rewards is known to be difficult as well as time-consuming. In this work, we address this problem by using natural language instructions to perform reward shaping. We propose the LanguagE-Action Reward Network (LEARN), a framework that maps free-form natural language instructions to intermediate rewards based on actions taken by the agent. These intermediate language-based rewards can seamlessly be integrated into any standard reinforcement learning algorithm. We experiment with Montezuma's Revenge from the Atari Learning Environment, a popular benchmark in RL. Our experiments on a diverse set of 15 tasks demonstrate that, for the same number of interactions with the environment, language-based rewards lead to successful completion of the task 60% more often on average, compared to learning without language.

Introduction

Reinforcement Learning (RL) has achieved significant milestones in mastering complex tasks, yet sample inefficiency remains a major hurdle. In environments with "sparse" rewards, such as the Atari game Montezuma's Revenge, an agent may explore for thousands of steps without receiving any feedback. Traditionally, "reward shaping"—adding intermediate rewards to guide the agent—requires manual, expert-level coding of mathematical functions. This paper proposes a more intuitive alternative: using natural language instructions to provide these rewards through the LanguagE-Action Reward Network (LEARN).

Methods

The LEARN framework acts as a bridge between human language and agent actions, operating in two distinct phases.

Overview of the LEARN framework for language-aided reinforcement learning.

First, the LEARN Module is trained to predict the relationship between a natural language instruction (e.g., "Jump over the skull") and an agent's trajectory. Trajectories are represented as Action-Frequency Vectors—histograms of actions taken over a period—rather than raw pixels, which improves computational efficiency.

Second, during the Language-Aided RL phase, LEARN provides a "shaping reward" whenever the agent's actions align with the provided instruction. To ensure the agent's optimal policy remains unchanged, the system utilizes Potential-Based Reward Shaping (PBRS), a mathematical technique that prevents the agent from "gaming" the intermediate rewards.

Results

The system was evaluated on 15 diverse tasks within Montezuma’s Revenge. Training data for LEARN included 20 human gameplay trajectories and 6,870 crowdsourced natural language descriptions, providing a realistic test against "noisy" input like typos and vague phrasing.

Comparison of learning performance across all tasks: Ext+Lang vs. ExtOnly baseline.

The experiments yielded several key insights:

  • Improved Success Rate: Agents using language-based rewards (Ext+Lang) completed tasks 60% more often than the baseline (ExtOnly) within 500,000 timesteps.
  • Efficiency Gains: The language-aided agents reached baseline performance levels with approximately 30% fewer environment interactions.
  • Robust Symbol Grounding: Despite the lack of visual state information, the model successfully mapped abstract terms like "climb" or "down" to specific internal action IDs, demonstrating that action distributions alone are surprisingly effective for grounding language.

Discussion and Limitations

While a significant step forward, the authors highlight areas for future development. The current use of action-frequency vectors ignores the temporal order of actions (e.g., "Jump then Left" vs "Left then Jump"). Additionally, the model is "state-blind," relying only on actions without visual context of the environment, which limits its ability to understand location-based instructions like "Go to the key." Future work aims to incorporate Recurrent Neural Networks (RNNs) for temporal awareness and multi-step instruction tracking.

By converting human speech into a mathematical reward signal, this research paves the way for a future where AI agents can be "programmed" through natural conversation.

Summary updated

Abstract

Recent reinforcement learning (RL) approaches have shown strong performance in complex domains such as Atari games, but are often highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the agent intermediate rewards for progress towards the goal. However, designing appropriate shaping rewards is known to be difficult as well as time-consuming. In this work, we address this problem by using natural language instructions to perform reward shaping. We propose the LanguagE-Action Reward Network (LEARN), a framework that maps free-form natural language instructions to intermediate rewards based on actions taken by the agent. These intermediate language-based rewards can seamlessly be integrated into any standard reinforcement learning algorithm. We experiment with Montezuma's Revenge from the Atari Learning Environment, a popular benchmark in RL. Our experiments on a diverse set of 15 tasks demonstrate that, for the same number of interactions with the environment, language-based rewards lead to successful completion of the task 60% more often on average, compared to learning without language.
BibTeX
@inproceedings{goyal2019using,
  author = {Prasoon Goyal and Scott Niekum and Raymond J. Mooney},
  title = {{Using Natural Language for Reward Shaping in Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=LIhl4IX5nV},
}
Figure