Large Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcement Learning

Guanwen Xie, Jingzehua Xu, Yiyuan Yang, Yimian Ding, Shuai Zhang

Abstract

Achieving the effective design and improvement of reward functions in reinforcement learning (RL) tasks with complex custom environments and multiple requirements presents considerable challenges. In this paper, we propose ERFSL, an efficient reward function searcher using LLMs, which enables LLMs to be effective white-box searchers and highlights their advanced semantic understanding capabilities. Specifically, we generate reward components for each numerically explicit user requirement and employ a reward critic to identify the correct code form. Then, LLMs assign weights to the reward components to balance their values and iteratively adjust the weights without ambiguity and redundant adjustments by flexibly adopting directional mutation and crossover strategies, similar to genetic algorithms, based on the context provided by the training log analyzer. We applied the framework to an underwater data collection RL task without direct human feedback or reward examples (zero-shot learning). The reward critic successfully corrects the reward code with only one feedback instance for each requirement, effectively preventing unrectifiable errors. The initialization of weights enables the acquisition of different reward functions within the Pareto solution set without the need for weight search. Even in cases where a weight is 500 times off, on average, only 5.2 iterations are needed to meet user requirements. The ERFSL also works well with most prompts utilizing GPT-4o mini, as we decompose the weight searching process to reduce the requirement for numerical and long-context understanding capabilities.

Introduction

In reinforcement learning (RL), the reward function serves as the primary mechanism for guiding agent behavior. However, designing effective reward functions is notoriously difficult, particularly in Multi-Objective Reinforcement Learning (MORL) scenarios. In these contexts, designers must balance competing objectives such as safety, efficiency, and performance. Poorly designed rewards often lead to "reward hacking," where agents exploit flaws in the function to achieve high scores without fulfilling the actual task requirements.

Traditional methods for reward design often involve tedious manual tuning or "black-box" optimization techniques that lack transparency and are difficult to debug. While Large Language Models (LLMs) have shown promise in code generation, applying them to reward design in complex, custom environments remains challenging due to the long-context nature of training logs and the precision required for numerical weight balancing.

The ERFSL Framework

To address these challenges, we introduce ERFSL (Efficient Reward Function Searcher using LLMs). ERFSL transforms the reward design process into a "white-box" search by decomposing the task into two distinct stages: reward code generation and reward weight optimization.

The ERFSL framework architecture, showing the interaction between the custom environment, code generator, critic, and weight searcher.

Stage 1: Reward Code Searcher

The first stage focuses on generating the logical structure of the reward function. Instead of generating a monolithic function, ERFSL prompts the LLM to create individual reward components for each specific user requirement (e.g., collision avoidance, energy efficiency).

A critical innovation in this stage is the Reward Critic. This specialized module reviews the generated code against the environment's variables and logic. If errors are detected—such as incorrect variable usage or flawed mathematical logic—the critic provides feedback, allowing the LLM to correct the code in a single iteration. This prevents "unrectifiable" bugs from propagating into the training phase.

Example of the Reward Critic correcting logic errors in the generated reward code to ensure proper collision avoidance behavior.

Stage 2: Reward Weight Searcher

Once the code is verified, the framework focuses on balancing the importance of each component. This stage involves three key sub-modules:

  1. Weight Initializer: The LLM estimates initial values to ensure all reward components are on a comparable numerical scale.
  2. Training Log Analyzer: This module summarizes noisy, high-dimensional training logs into concise textual descriptions. By converting numerical data into linguistic insights, it enables the LLM to reason about policy performance.
  3. Evolutionary Strategy: Inspired by genetic algorithms, the LLM performs "directional mutation" and "crossover" to iteratively adjust weights based on the analyzer's feedback.

Experimental Evaluation

We evaluated ERFSL in a custom Underwater Data Collection RL Environment. The task required multiple Autonomous Underwater Vehicles (AUVs) to collect data from sensors while avoiding collisions and minimizing energy consumption. The experiments were conducted in a zero-shot setting, providing the LLM with no prior examples of successful reward functions.

Key Results

  • Rapid Convergence: ERFSL demonstrated remarkable efficiency. Even when initial weights were intentionally set to be 500 times off the optimal value, the framework required an average of only 5.2 iterations to converge on a solution that met all user requirements.
  • One-Shot Correction: The Reward Critic consistently identified and fixed symbolic and logic errors with just one feedback instance per requirement, significantly reducing the manual debugging burden.
  • Model Robustness: While GPT-4o provided the highest reasoning capabilities, the decomposition of the search process allowed GPT-4o mini to achieve competitive results, making the framework accessible even with smaller models.

Conclusion

ERFSL represents a significant step toward automating the design of complex reward functions. By leveraging LLMs as white-box searchers and decomposing the problem into logical and numerical sub-tasks, the framework achieves high performance with minimal human intervention. Future work will focus on automating the initial environment description process and testing the framework's scalability across broader domains such as multi-agent robotics and power grid management.

Summary updated

Abstract

Achieving the effective design and improvement of reward functions in reinforcement learning (RL) tasks with complex custom environments and multiple requirements presents considerable challenges. In this paper, we propose ERFSL, an efficient reward function searcher using LLMs, which enables LLMs to be effective white-box searchers and highlights their advanced semantic understanding capabilities. Specifically, we generate reward components for each numerically explicit user requirement and employ a reward critic to identify the correct code form. Then, LLMs assign weights to the reward components to balance their values and iteratively adjust the weights without ambiguity and redundant adjustments by flexibly adopting directional mutation and crossover strategies, similar to genetic algorithms, based on the context provided by the training log analyzer. We applied the framework to an underwater data collection RL task without direct human feedback or reward examples (zero-shot learning). The reward critic successfully corrects the reward code with only one feedback instance for each requirement, effectively preventing unrectifiable errors. The initialization of weights enables the acquisition of different reward functions within the Pareto solution set without the need for weight search. Even in cases where a weight is 500 times off, on average, only 5.2 iterations are needed to meet user requirements. The ERFSL also works well with most prompts utilizing GPT-4o mini, as we decompose the weight searching process to reduce the requirement for numerical and long-context understanding capabilities
BibTeX
@misc{xie2024large,
  author = {Guanwen Xie and Jingzehua Xu and Yiyuan Yang and Yimian Ding and Shuai Zhang},
  title = {{Large Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcement Learning}},
  year = {2024},
  eprint = {2409.02428},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.02428},
}
Figure