ERFSL: An Efficient Reward Function Searcher via Large Language Models for Custom-Environment Multi-Objective Reinforcement Learning (Student Abstract)

Guanwen Xie, Jingzehua Xu, Yiyuan Yang, Yimian Ding, Shuai Zhang

Introduction

In Reinforcement Learning (RL), the reward function serves as the primary mechanism for defining agent behavior. However, designing effective reward functions for multi-objective tasks—where agents must balance competing goals such as speed, safety, and energy efficiency—remains a significant challenge. Human engineers often struggle with ambiguous requirements and the "weighting problem," where disparate reward components operate on vastly different scales, causing one objective to dominate the learning process.

While automated frameworks like Eureka have attempted to leverage Large Language Models (LLMs) for this task, they frequently produce logically flawed code or fail to balance weights effectively in complex, custom environments. ERFSL (Efficient Reward Function Searcher via LLMs) addresses these limitations by treating the LLM as a "white-box searcher" that utilizes semantic context to decompose reward design into manageable stages.

The ERFSL Framework

ERFSL employs a two-stage architecture designed to maximize both code quality and weight optimization efficiency.

Stage 1: Reward Code Generation

The process begins with a "meta-prompt" that helps the LLM refine high-level user requirements into specific numerical objectives. Rather than generating a single monolithic function, ERFSL generates individual reward components (e.g., safety, performance, energy). A "Reward Critic" module evaluates these components; if a logical flaw or bug is detected, it provides targeted feedback, typically allowing the LLM to rectify the code in a single iteration.

Once the code is finalized, ERFSL optimizes the weights of these components through two key innovations:

  1. Reward Weight Initializer (RWI): This tool pre-calculates component values to ensure they start on a comparable scale, solving the "cold start" problem where agents ignore small rewards due to dominant penalties.
  2. Iterative Optimization with TLA: A "Training Log Analyzer" (TLA) converts numerical RL training logs into textual summaries. The LLM then acts as a genetic algorithm, suggesting mutations (increasing or decreasing weights) and crossovers between weight groups based on its semantic understanding of the task performance.
The two-stage ERFSL architecture: Reward Code Generation and Reward Weight Search.

Experimental Evaluation

The framework was evaluated using a custom multi-AUV (Autonomous Underwater Vehicle) data collection environment. In this scenario, agents must navigate to target sensors, transfer data, avoid collisions, and minimize energy consumption. The Twin Delayed DDPG (TD3) algorithm was used for training.

Key Results

  • High Efficiency: With the Reward Weight Initializer, ERFSL achieved Pareto-optimal solutions in an average of only 0.4 iterations using GPT-4o.
  • Robustness: The system demonstrated remarkable resilience to initial conditions; even when starting weights were intentionally set 500x off the correct scale, the system converged in an average of 5.2 iterations.
  • Model Scaling: Smaller models like GPT-4o-mini, when integrated into the ERFSL architecture, outperformed larger baselines like EUREKA-M. This suggests that the framework effectively "boosts" the reasoning capabilities of smaller LLMs by reducing cognitive load through decomposition.
Performance evaluation showing Pareto-optimal solutions and convergence efficiency across different LLM backbones.

Discussion and Insights

A core insight of ERFSL is the value of Semantic Weighting. Unlike black-box optimizers that adjust numbers blindly, ERFSL uses the LLM's understanding of why a weight should change (e.g., "Energy consumption is too high, so increase the energy penalty"). This leads to more logical and faster convergence.

However, limitations remain. While GPT-4o-mini is highly capable, it lacks the flexibility in selecting "step sizes" for weight adjustments compared to the full GPT-4o model. Additionally, while the system is highly automated, the initial meta-prompting stage still benefits from human oversight to ensure high-quality task descriptions.

Conclusion

ERFSL provides a robust and efficient framework for automating reward function design in multi-objective RL. By splitting the problem into code generation and semantic weight search, it achieves superior performance with significantly fewer iterations than existing methods, making complex RL environment tuning more accessible and less error-prone.

Summary updated

Abstract

We propose ERFSL, an efficient reward function searcher using large language models (LLMs) for custom-environment, multi-objective reinforcement learning (RL). ERFSL generates reward components based on explicit user requirements and rectifies them, and iteratively optimizes the weights of these components based on textual context. Applied to an underwater data collection RL task, ERFSL corrects reward codes with only one feedback iteration per requirement, and acquires diverse reward functions within the Pareto set. ERFSL also presents robust capability for deviated weights and small-size LLMs such as GPT-4o mini. The full-text prompts, examples of LLM-generated answers, and source code are available at https://360zmem.github.io/LLMRsearcher/ .
BibTeX
@inproceedings{xie2025erfsl,
  author = {Guanwen Xie and Jingzehua Xu and Yiyuan Yang and Yimian Ding and Shuai Zhang},
  title = {{ERFSL: An Efficient Reward Function Searcher via Large Language Models for Custom-Environment Multi-Objective Reinforcement Learning (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35316},
}
Figure