Using Natural Language for Reward Shaping in Reinforcement Learning

Prasoon Goyal, Scott Niekum, Raymond J. Mooney

Abstract

Recent reinforcement learning (RL) approaches have shown strong performance in complex domains, such as Atari games, but are highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the agent intermediate rewards for progress towards the goal. Designing such rewards remains a challenge, though. In this work, we use natural language instructions to perform reward shaping. We propose a framework that maps free-form natural language instructions to intermediate rewards, that can seamlessly be integrated into any standard reinforcement learning algorithm. We experiment with Montezuma's Revenge from the Atari video games domain, a popular benchmark in RL. Our experiments on a diverse set of 15 tasks demonstrate that for the same number of interactions with the environment, using language-based rewards can successfully complete the task 60% more often, averaged across all tasks, compared to learning without language.

Introduction

In complex environments like the Atari game Montezuma’s Revenge, agents often face the sparse reward problem. An agent might wander for thousands of steps before accidentally stumbling upon a reward, such as picking up a key. Traditional solutions involve "reward shaping," where experts manually design intermediate rewards (e.g., points for moving toward a ladder). However, hand-coding these functions is tedious and requires domain expertise. This paper introduces LEARN (LanguagE-Action Reward Network), a framework that allows non-experts to provide guidance through simple, free-form natural language instructions, which the system automatically converts into a dense reward signal.

Methods

The LEARN framework employs a two-phase approach to bridge the gap between language and action.

The LEARN Model

LEARN is a neural network trained to predict the relevance of a sequence of actions to a specific natural language instruction.

The LEARN framework architecture, mapping language commands and action sequences to reward probabilities.
  • Action Representation: Instead of raw pixels, the model uses an action-frequency vector. This vector calculates the percentage of time the agent spent on each possible action (e.g., up, down, jump) during a specific time window.
  • Language Encoding: The authors evaluated multiple encoders, including InferSent (pretrained sentence embeddings), GloVe+RNN, and an RNN trained from scratch.
  • Output: The model outputs a probability score indicating how "related" the recent actions are to the instruction.

Language-Aided RL

This probability score is converted into a potential-based shaping reward. This mathematical formulation ensures that the added language rewards accelerate learning without altering the optimal policy defined by the original environment's rewards. The system was trained using human-played trajectories from the Atari Grand Challenge Dataset and 6,870 descriptions collected via Amazon Mechanical Turk.

Results and Key Insights

The integration of natural language instructions significantly improved agent performance across 15 diverse tasks, such as "go to the left and go under skulls."

Comparison of successful episodes over time for agents with (Ext+Lang) and without (ExtOnly) language-based reward shaping.
  • Significant Speedup: On average, the language-augmented agent completed tasks 60% more often than the baseline for the same number of environment interactions. Furthermore, it required 30% fewer timesteps to reach the baseline's peak performance.
  • Effective Grounding: The model successfully learned to associate words with actions. For example, the instruction "climb down the ladder" correctly correlated with "DOWN" actions and negatively correlated with "JUMP" or "RIGHT."
  • Robustness: By leveraging pretrained language models like InferSent, the system handled semantic variations (e.g., "jump" vs. "hop") effectively.

Discussion and Future Work

While LEARN demonstrates the power of language for reward shaping, it currently relies on action-frequency vectors, which lose the temporal order of actions. Future iterations could move toward full sequence modeling. Additionally, the current model does not incorporate visual state information; it knows "ladder" involves vertical movement but cannot visually identify a ladder. Integrating visual grounding would allow for more complex, state-dependent instructions. Finally, extending the framework to handle multi-step, sequential instructions remains a promising direction for future research.

Summary updated

Abstract

Recent reinforcement learning (RL) approaches have shown strong performance in complex domains, such as Atari games, but are highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the agent intermediate rewards for progress towards the goal. Designing such rewards remains a challenge, though. In this work, we use natural language instructions to perform reward shaping. We propose a framework that maps free-form natural language instructions to intermediate rewards, that can seamlessly be integrated into any standard reinforcement learning algorithm. We experiment with Montezuma's Revenge from the Atari video games domain, a popular benchmark in RL. Our experiments on a diverse set of 15 tasks demonstrate that for the same number of interactions with the environment, using language-based rewards can successfully complete the task 60% more often, averaged across all tasks, compared to learning without language.
BibTeX
@inproceedings{goyal2019using,
  author = {Prasoon Goyal and Scott Niekum and Raymond J. Mooney},
  title = {{Using Natural Language for Reward Shaping in Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=LIhl4IX5nV},
}
Figure