Multimodal Reward Shaping for Efficient Exploration in Reinforcement Learning

Mingqi Yuan, Mon-On Pun, Yi Chen, Dong Wang, Haojun LiarXiv 2021 · arXiv

Introduction

Reinforcement Learning (RL) agents face a fundamental challenge: the exploration-exploitation trade-off. To maximize long-term rewards, an agent must first discover them by exploring its environment. In sparse-reward or high-dimensional settings, standard exploration strategies often fail, causing agents to remain trapped in familiar regions of the state space. While intrinsic motivation—often termed "curiosity"—has been proposed to address this, existing methods frequently suffer from vanishing rewards or high computational complexity when estimating state visitation entropy from raw pixels.

This paper introduces Multimodal Reward Shaping (MMRS), a framework that reformulates exploration as a resource allocation problem. By leveraging Jain’s Fairness Index (JFI) as a global exploration metric and a Variational Auto-Encoder (VAE) for local novelty detection, MMRS provides a robust and efficient mechanism for driving agents toward undiscovered states.

The MMRS Framework

The core of MMRS is the decomposition of the intrinsic reward into two distinct components: a global "fairness" reward and a local "novelty" reward. This dual approach ensures that the agent is motivated both to visit entirely new regions and to distribute its time fairly across all known parts of the environment.

The MMRS architecture integrates a local intrinsic reward block based on VAE reconstruction and a global block using k-means clustering and Jain's Fairness Index.

Global Exploration: Jain’s Fairness Index

A significant contribution of this work is the application of Jain’s Fairness Index (JFI) to RL exploration. Originally used in networking to measure the equitable distribution of resources, JFI is applied here to state visitation. If an agent visits all states with equal frequency, the JFI is 1.0; if it focuses on a small subset, the index drops. The authors demonstrate that maximizing JFI is mathematically equivalent to maximizing state visitation entropy, but JFI offers superior stability and ease of calculation in practical applications.

To apply JFI in continuous or high-dimensional spaces, MMRS utilizes the latent representations from the VAE. These representations are clustered using k-means, effectively discretizing the environment into "bins" where visit counts can be tracked and used to compute the fairness score.

Local Novelty: VAE Reconstruction Error

While JFI handles global coverage, local novelty is managed by a VAE. The agent receives an intrinsic reward proportional to the reconstruction error of the current state. High error indicates a state the VAE has not yet learned to represent well, signaling novelty. This "life-long" curiosity prevents the agent from ignoring fine-grained details within larger clusters.

Experimental Results

The effectiveness of MMRS was evaluated across a diverse set of benchmarks, including tabular mazes, the Atari 2600 suite, and PyBullet continuous control tasks.

Efficiency in Maze Environments

In grid-based maze tasks, the inclusion of JFI significantly reduced the number of environment steps required to find the goal. As shown in the comparative results, the "Q-Learning + JFI" variant consistently outperformed standard Q-learning, with the performance gap widening as the maze size increased.

Minimum environment steps required to solve mazes of varying sizes, comparing standard Q-Learning against the JFI-enhanced version.

Performance in High-Dimensional Tasks

In the Atari 2600 benchmark, MMRS achieved state-of-the-art results, outperforming baselines like RND and RE3 in 16 out of 20 games. Notably, in games characterized by complex visual inputs and sparse rewards, MMRS showed more stable learning curves and higher final scores. Similarly, in the PyBullet suite, MMRS was the top performer across all six continuous control tasks, demonstrating its versatility across different action and observation spaces.

Conclusion

Multimodal Reward Shaping offers a compelling alternative to traditional entropy-based exploration. By reframing exploration through the lens of fairness and combining it with generative novelty detection, MMRS achieves high performance with greater computational efficiency. Future work may focus on dynamic clustering mechanisms to handle even larger environments and further refining the balance between global and local intrinsic rewards.

Summary updated

Abstract

Maintaining the long-term exploration capability of the agent remains one of the critical challenges in deep reinforcement learning. A representative solution is to leverage reward shaping to provide intrinsic rewards for the agent to encourage exploration. However, most existing methods suffer from vanishing intrinsic rewards, which cannot provide sustainable exploration incentives. Moreover, they rely heavily on complex models and additional memory to record learning procedures, resulting in high computational complexity and low robustness. To tackle this problem, entropy-based methods are proposed to evaluate the global exploration performance, encouraging the agent to visit the state space more equitably. However, the sample complexity of estimating the state visitation entropy is prohibitive when handling environments with high-dimensional observations. In this paper, we introduce a novel metric entitled Jain's fairness index (JFI) to replace the entropy regularizer, which solves the exploration problem from a brand new perspective. In sharp contrast to the entropy regularizer, JFI is more computable and robust and can be easily applied generalized into arbitrary tasks. Furthermore, we leverage a variational auto-encoder (VAE) model to capture the life-long novelty of states, which is combined with the global JFI score to form multimodal intrinsic rewards. Finally, extensive simulation results demonstrate that our multimodal reward shaping (MMRS) method can achieve higher performance than other benchmark schemes.
BibTeX
@misc{yuan2021multimodal,
  author = {Mingqi Yuan and Mon-On Pun and Yi Chen and Dong Wang and Haojun Li},
  title = {{Multimodal Reward Shaping for Efficient Exploration in Reinforcement Learning}},
  year = {2021},
  eprint = {2107.08888},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.08888},
}
Figure