Introduction
In Reinforcement Learning (RL), the reward function serves as the primary mechanism for defining agent behavior. However, designing effective reward functions for multi-objective tasks—where agents must balance competing goals such as speed, safety, and energy efficiency—remains a significant challenge. Human engineers often struggle with ambiguous requirements and the "weighting problem," where disparate reward components operate on vastly different scales, causing one objective to dominate the learning process.
While automated frameworks like Eureka have attempted to leverage Large Language Models (LLMs) for this task, they frequently produce logically flawed code or fail to balance weights effectively in complex, custom environments. ERFSL (Efficient Reward Function Searcher via LLMs) addresses these limitations by treating the LLM as a "white-box searcher" that utilizes semantic context to decompose reward design into manageable stages.
The ERFSL Framework
ERFSL employs a two-stage architecture designed to maximize both code quality and weight optimization efficiency.
Stage 1: Reward Code Generation
The process begins with a "meta-prompt" that helps the LLM refine high-level user requirements into specific numerical objectives. Rather than generating a single monolithic function, ERFSL generates individual reward components (e.g., safety, performance, energy). A "Reward Critic" module evaluates these components; if a logical flaw or bug is detected, it provides targeted feedback, typically allowing the LLM to rectify the code in a single iteration.
Stage 2: Reward Weight Search
Once the code is finalized, ERFSL optimizes the weights of these components through two key innovations:
- Reward Weight Initializer (RWI): This tool pre-calculates component values to ensure they start on a comparable scale, solving the "cold start" problem where agents ignore small rewards due to dominant penalties.
- Iterative Optimization with TLA: A "Training Log Analyzer" (TLA) converts numerical RL training logs into textual summaries. The LLM then acts as a genetic algorithm, suggesting mutations (increasing or decreasing weights) and crossovers between weight groups based on its semantic understanding of the task performance.
Experimental Evaluation
The framework was evaluated using a custom multi-AUV (Autonomous Underwater Vehicle) data collection environment. In this scenario, agents must navigate to target sensors, transfer data, avoid collisions, and minimize energy consumption. The Twin Delayed DDPG (TD3) algorithm was used for training.
Key Results
- High Efficiency: With the Reward Weight Initializer, ERFSL achieved Pareto-optimal solutions in an average of only 0.4 iterations using GPT-4o.
- Robustness: The system demonstrated remarkable resilience to initial conditions; even when starting weights were intentionally set 500x off the correct scale, the system converged in an average of 5.2 iterations.
- Model Scaling: Smaller models like GPT-4o-mini, when integrated into the ERFSL architecture, outperformed larger baselines like EUREKA-M. This suggests that the framework effectively "boosts" the reasoning capabilities of smaller LLMs by reducing cognitive load through decomposition.
Discussion and Insights
A core insight of ERFSL is the value of Semantic Weighting. Unlike black-box optimizers that adjust numbers blindly, ERFSL uses the LLM's understanding of why a weight should change (e.g., "Energy consumption is too high, so increase the energy penalty"). This leads to more logical and faster convergence.
However, limitations remain. While GPT-4o-mini is highly capable, it lacks the flexibility in selecting "step sizes" for weight adjustments compared to the full GPT-4o model. Additionally, while the system is highly automated, the initial meta-prompting stage still benefits from human oversight to ensure high-quality task descriptions.
Conclusion
ERFSL provides a robust and efficient framework for automating reward function design in multi-objective RL. By splitting the problem into code generation and semantic weight search, it achieves superior performance with significantly fewer iterations than existing methods, making complex RL environment tuning more accessible and less error-prone.
Summary updated