Introduction
Reinforcement Learning (RL) agents face a fundamental challenge: the exploration-exploitation trade-off. To maximize long-term rewards, an agent must first discover them by exploring its environment. In sparse-reward or high-dimensional settings, standard exploration strategies often fail, causing agents to remain trapped in familiar regions of the state space. While intrinsic motivation—often termed "curiosity"—has been proposed to address this, existing methods frequently suffer from vanishing rewards or high computational complexity when estimating state visitation entropy from raw pixels.
This paper introduces Multimodal Reward Shaping (MMRS), a framework that reformulates exploration as a resource allocation problem. By leveraging Jain’s Fairness Index (JFI) as a global exploration metric and a Variational Auto-Encoder (VAE) for local novelty detection, MMRS provides a robust and efficient mechanism for driving agents toward undiscovered states.
The MMRS Framework
The core of MMRS is the decomposition of the intrinsic reward into two distinct components: a global "fairness" reward and a local "novelty" reward. This dual approach ensures that the agent is motivated both to visit entirely new regions and to distribute its time fairly across all known parts of the environment.
Global Exploration: Jain’s Fairness Index
A significant contribution of this work is the application of Jain’s Fairness Index (JFI) to RL exploration. Originally used in networking to measure the equitable distribution of resources, JFI is applied here to state visitation. If an agent visits all states with equal frequency, the JFI is 1.0; if it focuses on a small subset, the index drops. The authors demonstrate that maximizing JFI is mathematically equivalent to maximizing state visitation entropy, but JFI offers superior stability and ease of calculation in practical applications.
To apply JFI in continuous or high-dimensional spaces, MMRS utilizes the latent representations from the VAE. These representations are clustered using k-means, effectively discretizing the environment into "bins" where visit counts can be tracked and used to compute the fairness score.
Local Novelty: VAE Reconstruction Error
While JFI handles global coverage, local novelty is managed by a VAE. The agent receives an intrinsic reward proportional to the reconstruction error of the current state. High error indicates a state the VAE has not yet learned to represent well, signaling novelty. This "life-long" curiosity prevents the agent from ignoring fine-grained details within larger clusters.
Experimental Results
The effectiveness of MMRS was evaluated across a diverse set of benchmarks, including tabular mazes, the Atari 2600 suite, and PyBullet continuous control tasks.
Efficiency in Maze Environments
In grid-based maze tasks, the inclusion of JFI significantly reduced the number of environment steps required to find the goal. As shown in the comparative results, the "Q-Learning + JFI" variant consistently outperformed standard Q-learning, with the performance gap widening as the maze size increased.
Performance in High-Dimensional Tasks
In the Atari 2600 benchmark, MMRS achieved state-of-the-art results, outperforming baselines like RND and RE3 in 16 out of 20 games. Notably, in games characterized by complex visual inputs and sparse rewards, MMRS showed more stable learning curves and higher final scores. Similarly, in the PyBullet suite, MMRS was the top performer across all six continuous control tasks, demonstrating its versatility across different action and observation spaces.
Conclusion
Multimodal Reward Shaping offers a compelling alternative to traditional entropy-based exploration. By reframing exploration through the lens of fairness and combining it with generative novelty detection, MMRS achieves high performance with greater computational efficiency. Future work may focus on dynamic clustering mechanisms to handle even larger environments and further refining the balance between global and local intrinsic rewards.
Summary updated