Discovery of the reward function for embodied reinforcement learning agents

Renzhi Lu, Zonghe Shao, Yuemin Ding, Ruijuan Chen, Dongrui Wu, Housheng Su, Tao Yang, Fumin Zhang, Jun Wang, Yang Shi, Zhong‐Ping Jiang, Ding Han, Haitao ZhangNature Communications 2025 · DOI

Abstract

Reward maximization is a fundamental principle in both the survival and evolution of biological organisms. In particular, in the contexts of cognitive science and embodied agents, reward-driven behavior has been widely regarded as important in terms of governing complex cognitive abilities such as perception, imitation, and learning. Among the frameworks that are aimed at establishing such abilities, reinforcement learning (RL), which leverages reward maximization to facilitate intelligent decision-making in embodied agents, has been proven to be particularly promising. Importantly, the inherent complexity and uncertainty of real-world tasks pose significant challenges when designing effective reward functions for embodied RL agents. Conventional methods typically rely on manually engineered or externally tuned reward signals, and therefore require significant domain expertise, associated with considerable human efforts and a long convergence time; these issues may even trigger mission failure. This work introduces a bilevel optimization framework that discovers optimal reward functions for embodied reinforcement learning agents through a mechanism called regret minimization. The approach accelerates policy optimization and enhances adaptability across diverse tasks. These findings can support the broader adoption of embodied RL agents in the behavioral and computational sciences and neurosciences, thereby paving the way for artificial general intelligence.

Introduction

In both biological and artificial systems, intelligence is often defined by the ability to maximize rewards within an environment. Natural intelligence has evolved over millennia to optimize for survival and reproduction, resulting in complex traits like adaptation and creativity. Embodied artificial intelligence seeks to mirror this process, using reinforcement learning (RL) to develop agents capable of sophisticated decision-making.

Conceptual parallel between natural intelligence evolution and the development of embodied intelligence through reward maximization.

However, a significant bottleneck remains: the reward function. Traditionally, human designers must manually engineer these signals. As task complexity increases, the human effort required to design effective rewards grows exponentially, often leading to "reward hacking" or slow convergence.

The Challenge of Reward Engineering

Reinforcement learning's "Achilles' heel" is the reliance on dense, well-shaped reward functions. If a reward is too sparse (e.g., a simple success/fail signal), an agent may never learn. Conversely, poorly designed rewards can lead to the "cobra effect," where agents find unintended shortcuts to maximize points without solving the actual task. While Inverse Reinforcement Learning (IRL) attempts to address this by learning from experts, it remains constrained by the availability and cost of high-quality demonstrations.

Proposed Framework: Bilevel Optimization and Regret Minimization

This research introduces a bilevel optimization framework that treats reward discovery as a meta-learning problem, allowing agents to autonomously "discover" optimal reward landscapes with minimal prior knowledge.

Overview of the reward discovery framework, illustrating the challenge of human effort (a, b), the regret minimization objective (c), the bilevel optimization architecture (d), and the transition from initialized to optimal reward landscapes (e, f).

The Bilevel Loop

  1. Lower Level (Policy Optimization): The agent interacts with a world model to maximize a parameterized "discovered" reward function ($R_\omega$).
  2. Upper Level (Reward Optimization): This level evaluates the reward function's effectiveness using regret minimization. Regret is defined as the performance gap between the agent's current policy and a theoretical optimal policy, measured against a sparse, high-level success signal.
  3. Meta-Gradients: A novel meta-gradient algorithm updates the reward parameters ($\omega$) by leveraging the advantage function. This ensures compatibility with various RL architectures, including DQN, PPO, and SAC.

Experimental Evaluation and Results

The framework was tested across diverse environments, from classic control tasks to high-dimensional physical simulations and real-world industrial applications:

  • High-Dimensional Control (MuJoCo): In complex tasks like Humanoid-v2, the discovered rewards led to faster convergence and higher final returns than manually designed rewards or existing state-of-the-art baselines.
  • Data Center Energy Management: Using real-world data, RL agents guided by discovered rewards achieved energy reductions exceeding 60%, significantly outperforming standard RL models.
  • UAV Precision Navigation: The framework successfully captured "critical states" (such as imminent crashes) that traditional dynamic principles often miss, leading to superior waypoint navigation for quadrotor drones.

Key Insights

  • Density from Sparsity: The system transforms binary success signals into dense, informative landscapes that guide the agent at every step.
  • Emergent Physics: In tasks like the Acrobot, the discovered reward naturally prioritized actions that built momentum, effectively "discovering" the physics of energy conservation without explicit programming.
  • Stability: Synchronous updates of the reward and policy provide a more stable learning trajectory than traditional trial-and-error reward shaping.

Conclusion and Future Work

By automating the discovery of reward functions, this framework reduces the need for domain expertise and manual engineering in RL. While currently dependent on world models and computationally intensive due to nested optimization, this approach represents a significant step toward more generalized and adaptable artificial intelligence. Future research will explore model-free applications and zero-shot reward transfer across different robotic morphologies.

Summary updated

Abstract

Reward maximization is a fundamental principle in both the survival and evolution of biological organisms. In particular, in the contexts of cognitive science and embodied agents, reward-driven behavior has been widely regarded as important in terms of governing complex cognitive abilities such as perception, imitation, and learning. Among the frameworks that are aimed at establishing such abilities, reinforcement learning (RL), which leverages reward maximization to facilitate intelligent decision-making in embodied agents, has been proven to be particularly promising. Importantly, the inherent complexity and uncertainty of real-world tasks pose significant challenges when designing effective reward functions for embodied RL agents. Conventional methods typically rely on manually engineered or externally tuned reward signals, and therefore require significant domain expertise, associated with considerable human efforts and a long convergence time; these issues may even trigger mission failure. This work introduces a bilevel optimization framework that discovers optimal reward functions for embodied reinforcement learning agents through a mechanism called regret minimization. The approach accelerates policy optimization and enhances adaptability across diverse tasks. These findings can support the broader adoption of embodied RL agents in the behavioral and computational sciences and neurosciences, thereby paving the way for artificial general intelligence.
BibTeX
@inproceedings{lu2025discovery,
  author = {Renzhi Lu and Zonghe Shao and Yuemin Ding and Ruijuan Chen and Dongrui Wu and Housheng Su and Tao Yang and Fumin Zhang and Jun Wang and Yang Shi and Zhong‐Ping Jiang and Ding Han and Haitao Zhang},
  title = {{Discovery of the reward function for embodied reinforcement learning agents}},
  booktitle = {Nature Communications},
  year = {2025},
  doi = {10.1038/s41467-025-66009-y},
}
Figure