Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos

Harsh Mahesheka, Zhixian Xie, Zhaoran Wang, Wanxin JinarXiv 2024 · DOI · arXiv

Abstract

Learning complex robot behaviors from raw internet videos remains a significant challenge due to morphological differences and the need for labor-intensive pose estimation and motion retargeting. We present a novel bi-level programming framework that leverages the semantic reasoning of Vision-Language Models (VLMs) and Large Language Models (LLMs) to automate reward function design. By utilizing Gemini 1.5 Pro for visual critique and GPT-4o for reward code generation, our system enables robots to mimic biological motions directly from YouTube clips without manual preprocessing. Our results in the Isaac Gym simulator demonstrate that this semantic, gradient-free approach outperforms existing state-of-the-art LLM-based reward design methods and achieves performance comparable to human-led feedback loops.

1. Introduction

Learning from demonstrations (LfD) is a cornerstone of modern robotics, yet the transition from raw video data to functional robot policies is often hindered by technical bottlenecks. Traditional pipelines require precise pose estimation, complex mapping between disparate anatomies (e.g., from a spider to a robot ant), and the manual engineering of reward functions. These steps are brittle, task-specific, and difficult to scale.

This paper introduces a framework designed to bypass these intermediate steps by treating reward learning as a bi-level optimization problem solved through linguistic reasoning. By replacing mathematical derivatives with semantic feedback from foundation models, we enable robots to learn "in-the-wild" behaviors directly from internet videos.

2. Methodology

2.1 Bi-Level Framework Overview

The proposed system operates as a gradient-free bi-level loop that mimics the chain rule of optimization. The process begins with a robot policy trained on an initial reward function. The resulting behavior is recorded and compared against an expert video.

Overview of the bi-level reward learning framework integrating VLMs and LLMs.

2.2 Upper-Level: Visual Critique

In the upper level, a Vision-Language Model (Gemini 1.5 Pro) acts as the visual critic. It is presented with two inputs: the expert demonstration (e.g., a jumping spider) and the robot's current attempt in the simulator. The VLM identifies discrepancies between the two, providing natural language feedback that highlights specific failures, such as a lack of vertical displacement or insufficient crouching before a leap.

Visual feedback generated by the Upper-Level VLM (Gemini) comparing robot behavior to expert video.

2.3 Lower-Level: Reward Engineering

The visual feedback is passed to the lower level, where a Large Language Model (GPT-4o) serves as the reward engineer. Using the text feedback and the robot's environment code, the LLM generates or updates a Python-based reward function. This "reward as code" approach allows the system to incorporate complex physical concepts—such as stability penalties or energy storage—that are difficult to capture with simple scalar values.

3. Experimental Evaluation

3.1 Setup

The framework was tested in the Isaac Gym simulation environment using diverse "in-the-wild" video clips from YouTube and Pexels. Tasks included an "Ant" robot mimicking jumping spiders, a "Humanoid" performing splits and running, and an "ANYmal" quadruped mimicking hopping dogs.

3.2 Key Results

  • Performance vs. State-of-the-Art: The system was compared against Eureka, a leading LLM-based reward design method. Our framework consistently achieved higher Normalized Expert Scores. The primary advantage lies in the visual feedback loop; while Eureka relies on textual environment descriptions, our method "sees" the robot's motion, allowing for more precise adjustments.
  • Human-Level Feedback: In baseline tests where human experts provided the visual critique instead of Gemini, the VLM-based system performed nearly as well, indicating that modern foundation models can provide feedback of sufficient quality for complex mimicry.
  • Biological Realism: Qualitatively, the robots developed nuanced behaviors. For instance, the Ant robot learned to crouch and store potential energy before jumping, a direct mimicry of the biological motion pattern observed in the spider videos.

4. Discussion and Conclusion

By shifting from analytical pose-matching to semantic reasoning, this framework offers a robust path for robots to learn from the vast library of human and animal behavior available online. However, limitations remain. The system is sensitive to visual noise (e.g., multiple agents in a video) and involves high computational costs due to the iterative nature of RL training and API-based model calls. Future work will focus on improving efficiency and demonstrating these learned rewards on physical hardware to address the sim-to-real gap.

Summary updated

Abstract

Learning from Demonstrations, particularly from biological experts like humans and animals, often encounters significant data acquisition challenges. While recent approaches leverage internet videos for learning, they require complex, task-specific pipelines to extract and retarget motion data for the agent. In this work, we introduce a language-model-assisted bi-level programming framework that enables a reinforcement learning agent to directly learn its reward from internet videos, bypassing dedicated data preparation. The framework includes two levels: an upper level where a vision-language model (VLM) provides feedback by comparing the learner's behavior with expert videos, and a lower level where a large language model (LLM) translates this feedback into reward updates. The VLM and LLM collaborate within this bi-level framework, using a "chain rule" approach to derive a valid search direction for reward learning. We validate the method for reward learning from YouTube videos, and the results have shown that the proposed method enables efficient reward design from expert videos of biological agents for complex behavior synthesis.
BibTeX
@misc{mahesheka2024language,
  author = {Harsh Mahesheka and Zhixian Xie and Zhaoran Wang and Wanxin Jin},
  title = {{Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos}},
  year = {2024},
  eprint = {2410.09286},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.09286},
}
Figure