Abstract
TGRPO (Trajectory-wise Group Relative Policy Optimization) is a novel reinforcement learning framework for fine-tuning Vision-Language-Action (VLA) models. Unlike traditional RL approaches that require complex critic networks, TGRPO utilizes a group-based relative comparison strategy. By integrating LLM-designed multi-stage rewards and a dual-level advantage fusion mechanism that balances global trajectory progress with local step consistency, TGRPO significantly improves robotic performance on complex, long-horizon tasks, outperforming standard imitation learning baselines.
Introduction: The Generalization Gap in Robotics
Most state-of-the-art robotic models (VLAs) are trained using Supervised Fine-Tuning (SFT) on human demonstrations. While effective, this approach often leads to "action memorization"—the robot learns to mimic a specific path rather than understanding the underlying logic of the environment. Consequently, if the robot encounters a slightly new scenario (Out-of-Distribution) or makes a small execution error, it cannot self-correct and fails.
Reinforcement Learning (RL) offers a solution through trial-and-error, but applying it to large-scale VLA models is computationally expensive. Standard RL algorithms like PPO require training a second "Critic" network to estimate values, which is difficult for large models. Furthermore, robotic tasks often have "sparse rewards," making it hard for the robot to credit specific actions for final success.
Methods: Critic-Free Group Optimization
The authors introduce TGRPO, a "critic-free" training framework that replaces the traditional value network with a group-based relative comparison strategy.
1. LLM-Guided Multi-Stage Rewards
To overcome the sparse reward problem, the researchers used Claude 3.7 Sonnet to automatically decompose complex natural language instructions (e.g., "put the soup in the basket") into sub-stages like approaching, grasping, and placing. The LLM generates dense reward functions based on the spatial distance between the robot's gripper and key objects.
2. Group Relative Optimization
Instead of comparing an action against a predicted "value," TGRPO samples multiple trajectories in parallel. It then calculates how much better or worse a specific trajectory (or step) is compared to the average of that group. This eliminates the need for a memory-intensive Critic network.
3. Dual-Level Advantage Fusion
TGRPO calculates two types of advantages:
- Trajectory-level: Captures global success (did we finish the whole task?).
- Step-level: Captures local consistency (was this specific movement smooth and goal-oriented?).
By fusing these signals, the model avoids getting stuck on intermediate sub-goals while maintaining high-precision control.
Experimental Setup and Datasets
The model was evaluated on the LIBERO benchmark, a standard suite for lifelong robotic learning. It includes four task categories: LIBERO-Spatial (spatial relationships), LIBERO-Object (object categories), LIBERO-Goal (diverse task goals), and LIBERO-Long (long-horizon, multi-step manipulation).
Key Results
TGRPO demonstrated significant improvements over existing methods:
- Superior Success Rates: TGRPO achieved an average success rate of 80.7% across all LIBERO tasks, outperforming the SFT baseline (76.5%) and other RL methods like GRAPE (79.2%).
- Dominance in Long-Horizon Tasks: On the "LIBERO-Long" suite, TGRPO outperformed SFT by 8.1%, highlighting its ability to handle multi-stage reasoning.
Discussion and Future Work
TGRPO proves that the "Critic" is optional in large-scale VLA fine-tuning. By using group-relative comparisons and LLM-engineered rewards, the framework reduces memory overhead and improves training stability.
However, challenges remain. The current study was conducted entirely within the LIBERO simulator; the "sim-to-real" gap involving real-world sensor noise and physics is a key area for future research. Additionally, the authors aim to extend TGRPO to multi-task settings where a single policy can handle a vast array of diverse instructions simultaneously.
Summary updated