Abstract
Learning complex robot behaviors from raw internet videos remains a significant challenge due to morphological differences and the need for labor-intensive pose estimation and motion retargeting. We present a novel bi-level programming framework that leverages the semantic reasoning of Vision-Language Models (VLMs) and Large Language Models (LLMs) to automate reward function design. By utilizing Gemini 1.5 Pro for visual critique and GPT-4o for reward code generation, our system enables robots to mimic biological motions directly from YouTube clips without manual preprocessing. Our results in the Isaac Gym simulator demonstrate that this semantic, gradient-free approach outperforms existing state-of-the-art LLM-based reward design methods and achieves performance comparable to human-led feedback loops.
1. Introduction
Learning from demonstrations (LfD) is a cornerstone of modern robotics, yet the transition from raw video data to functional robot policies is often hindered by technical bottlenecks. Traditional pipelines require precise pose estimation, complex mapping between disparate anatomies (e.g., from a spider to a robot ant), and the manual engineering of reward functions. These steps are brittle, task-specific, and difficult to scale.
This paper introduces a framework designed to bypass these intermediate steps by treating reward learning as a bi-level optimization problem solved through linguistic reasoning. By replacing mathematical derivatives with semantic feedback from foundation models, we enable robots to learn "in-the-wild" behaviors directly from internet videos.
2. Methodology
2.1 Bi-Level Framework Overview
The proposed system operates as a gradient-free bi-level loop that mimics the chain rule of optimization. The process begins with a robot policy trained on an initial reward function. The resulting behavior is recorded and compared against an expert video.
2.2 Upper-Level: Visual Critique
In the upper level, a Vision-Language Model (Gemini 1.5 Pro) acts as the visual critic. It is presented with two inputs: the expert demonstration (e.g., a jumping spider) and the robot's current attempt in the simulator. The VLM identifies discrepancies between the two, providing natural language feedback that highlights specific failures, such as a lack of vertical displacement or insufficient crouching before a leap.
2.3 Lower-Level: Reward Engineering
The visual feedback is passed to the lower level, where a Large Language Model (GPT-4o) serves as the reward engineer. Using the text feedback and the robot's environment code, the LLM generates or updates a Python-based reward function. This "reward as code" approach allows the system to incorporate complex physical concepts—such as stability penalties or energy storage—that are difficult to capture with simple scalar values.
3. Experimental Evaluation
3.1 Setup
The framework was tested in the Isaac Gym simulation environment using diverse "in-the-wild" video clips from YouTube and Pexels. Tasks included an "Ant" robot mimicking jumping spiders, a "Humanoid" performing splits and running, and an "ANYmal" quadruped mimicking hopping dogs.
3.2 Key Results
- Performance vs. State-of-the-Art: The system was compared against Eureka, a leading LLM-based reward design method. Our framework consistently achieved higher Normalized Expert Scores. The primary advantage lies in the visual feedback loop; while Eureka relies on textual environment descriptions, our method "sees" the robot's motion, allowing for more precise adjustments.
- Human-Level Feedback: In baseline tests where human experts provided the visual critique instead of Gemini, the VLM-based system performed nearly as well, indicating that modern foundation models can provide feedback of sufficient quality for complex mimicry.
- Biological Realism: Qualitatively, the robots developed nuanced behaviors. For instance, the Ant robot learned to crouch and store potential energy before jumping, a direct mimicry of the biological motion pattern observed in the spider videos.
4. Discussion and Conclusion
By shifting from analytical pose-matching to semantic reasoning, this framework offers a robust path for robots to learn from the vast library of human and animal behavior available online. However, limitations remain. The system is sensitive to visual noise (e.g., multiple agents in a video) and involves high computational costs due to the iterative nature of RL training and API-based model calls. Future work will focus on improving efficiency and demonstrating these learned rewards on physical hardware to address the sim-to-real gap.
Summary updated