Evolutionary Reward Design and Optimization with Multimodal Large Language Models

Ali NarinACL 2024

TL;DR

This paper introduces EROM, a novel method that leverages multimodal large language models (MLLMs) to automatically design and optimize reward functions for reinforcement learning (RL). By incorporating visual feedback (agent videos and idle images) alongside traditional textual and numerical data, EROM significantly improves reward function performance compared to previous LLM-based approaches.

What Problem Does This Paper Address?

Designing effective reward functions is a notoriously difficult and time-consuming task in reinforcement learning. It often requires deep domain expertise and extensive manual tuning. While recent work has shown promise in using LLMs to automate this process through evolutionary search and textual feedback, a significant limitation remains: LLMs typically struggle with non-textual data. Human experts, however, naturally use visual cues to understand agent behavior and identify areas for improvement. This paper addresses the gap of incorporating visual information into the automated reward design pipeline, hypothesizing that it will lead to more comprehensive understanding and, consequently, better reward functions.

Methods

The core of the proposed method, EROM (Evolutionary Reward Design and Optimization with Multimodal Large Language Models), builds upon the "Eureka" method, which uses LLMs and evolutionary search. The key innovation is the integration of multimodal capabilities.

Here's how EROM works:

  1. Environment as Context: The MLLM (specifically GPT-4V) is provided with the RL environment's source code to understand its mechanics.
  2. Initial Reward Generation: The MLLM is also given a description of the task and an image of the agent in its idle state. This initial visual input helps the MLLM understand the agent's basic form and context.
  3. Evolutionary Search & Feedback Loop:
    • The MLLM generates an initial set of reward function candidates.
  4. These candidates are tested in the RL environment, and the top-performing agent's behavior is recorded.
  5. Crucially, the feedback provided to the MLLM for the next iteration includes both the numerical performance metrics (e.g., success rate, episode length) and a video of the top-performing agent's training.
  6. The MLLM then uses this multimodal feedback to iteratively refine and improve the reward function.

This iterative process of "reward reflection" allows the MLLM to learn from the agent's visual performance, identifying potential issues like reward hacking or suboptimal strategies that might be missed with purely textual feedback.

Datasets

The experiments were conducted using the IsaacGYM simulator. The paper focuses on two specific agents and tasks within this simulator:

  • Ant Agent: Tasked with running forward as fast as possible.
  • Humanoid Agent: Also tasked with running forward as fast as possible.

The paper does not introduce new datasets but rather uses these simulated environments to evaluate the effectiveness of their reward design method.

Key Insights and Ideas

  • Visual Feedback is Crucial: The most significant insight is that incorporating visual feedback (agent videos) into the LLM's reward design process substantially improves performance compared to relying solely on textual and numerical data. This validates the hypothesis that visual cues provide a richer understanding of agent behavior.
  • MLLMs Excel at Multimodal Reward Design: GPT-4V's ability to process both images and videos, combined with its coding capabilities, makes it a powerful tool for this complex task.
  • Evolutionary Search Amplifies Gains: The iterative nature of evolutionary search, guided by multimodal feedback, allows for continuous refinement and optimization of reward functions, leading to progressively better outcomes.
  • Beyond Textual Limitations: This work demonstrates a clear path to overcoming the inherent limitations of LLMs in handling non-textual information, opening doors for more sophisticated AI-driven design processes.

Key Results

The EROM method demonstrated superior performance across various experimental setups:

  • General Testing (5 iterations, 8 samples/iteration): EROM achieved an average success rate of 7.27 for the ant agent and 5.26 for the humanoid agent. This significantly outperformed the baseline "Eureka" method, which achieved 3.68 and 4.21, respectively.
  • Image Testing (1 iteration, 32 samples): Providing an idle agent image in the first generation improved the average success rate for the ant agent.
  • Video Testing (32 samples): When comparing feedback mechanisms, EROM with video feedback consistently improved the average success rate for both agents compared to methods using only numerical feedback.

These results indicate that EROM's approach is more effective than previous methods, which themselves outperformed 83% of human-expert-designed reward functions.

Limitations and Future Work

The authors acknowledge several limitations:

  • Reliance on GPT-4V Capabilities: The performance of EROM is inherently tied to the capabilities of the specific MLLM used.
  • Simulation vs. Real World: The success in simulation might not directly translate to the complexities of real-world robotic applications.
  • Limited Agent Count: Due to GPU memory constraints, experiments were limited to two agents. Testing with more agents and diverse environments would better showcase generalization.

Future work could involve exploring these limitations, such as testing on more complex real-world robotic tasks and evaluating the method with a wider range of agents and environments. The authors also suggest that further research could investigate how to best leverage the visual information for even more nuanced reward design.

Summary updated

BibTeX
@inproceedings{narin2024evolutionary,
  author = {Ali Narin},
  title = {{Evolutionary Reward Design and Optimization with Multimodal Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure