Abstract: Reward functions in reinforcement learning have largely been assumed given as part of the problem being solved by the agent. However, the psychological notion of intrinsic motivation has recently inspired inquiry into whether there exist alternate reward functions that enable an agent to learn a task more easily than the natural task-based reward function allows. This paper presents an efficient genetic programming algorithm to search for alternate reward functions that improve agent learning performance.
Introduction
In traditional reinforcement learning (RL), agents are typically guided by an external reward function tied directly to a final objective—for instance, receiving a point only upon reaching a goal. However, biological systems often rely on internal motivational systems. These "intrinsic" rewards, such as curiosity or hunger satisfaction, guide learning long before a primary goal is achieved.
While previous research has demonstrated that alternate reward functions can significantly accelerate learning, finding these functions has historically relied on brute-force searches over small, discretized variable sets. This paper addresses the challenge of efficiently searching the vast space of possible reward functions to identify those that generalize across environments.
Methodology: Evolving Rewards with PushGP
The core of this approach is the use of Genetic Programming (GP), specifically the PushGP system. Unlike standard GP that evolves tree structures, PushGP is a stack-based evolutionary system capable of searching the space of all computable programs. This expressiveness allows for the discovery of complex, continuous reward functions.
In this evolutionary process, a population of reward functions is tested across a distribution of environments. The "fitness" of a specific reward function is determined by the cumulative success of an agent using that function to achieve an ultimate goal (e.g., gathering food). The evolved functions can take various inputs, including internal states (hunger/thirst), spatial coordinates (X, Y position), and even irrelevant "distractor" variables to test robustness.
The Hungry-Thirsty Domain
The methodology was evaluated using the Hungry-Thirsty domain, a grid-world benchmark. In this setup, food and water are placed at random corners. The agent's goal is to eat to gain fitness points, but it cannot eat while thirsty. Drinking makes the agent "not-thirsty" for one step, after which there is a 10% chance of becoming thirsty again. This creates a complex dependency where the agent must manage internal physiological states alongside spatial navigation.
Key Results and Insights
The experimental results demonstrate that evolved reward functions significantly outperform standard baselines and previous state-of-the-art search methods.
- Superior Performance: Agents using evolved rewards achieved higher cumulative fitness over their lifetimes compared to those using standard fitness-based rewards or rewards found via brute-force search.
- Leveraging Information: The 4-variable reward (which included spatial X/Y coordinates) eventually outperformed the 2-variable version, showing that the evolutionary process can effectively extract and utilize salient features from the environment.
- Robustness to Noise: A striking finding was the system's resilience to "distractor" variables. As shown in the evolutionary progress plots, adding random noise barely hindered the discovery of effective reward functions, suggesting the method can scale to high-dimensional real-world data.
Conclusion and Future Work
By moving from discretized brute-force searches to continuous evolutionary programming, this work provides a scalable path for discovering intrinsic motivations. The evolved rewards do more than just solve a single map; they capture common strategic features across a distribution of environments.
Future research may explore using these evolved rewards for autonomous skill identification. If a reward function captures a common sub-task—such as seeking water when thirsty—it could help agents hierarchically decompose complex problems or facilitate transfer learning between related tasks. While computationally intensive, the ability to "pre-calculate" effective learning strategies through evolution offers a powerful tool for developing more capable autonomous agents.
Summary updated