TL;DR
PGX (Physically Grounded Execution) is a modular framework that enables humanoid robots to master long-horizon tasks by combining modern Vision-Language-Action (VLA) models with classical, training-free control layers. By offloading task timing and spatial precision to a state machine and a coordinate-locking system, PGX turns "near-failing" end-to-end policies into reliable systems capable of complex sequences like fetching and brewing coffee.
What Problem Does This Paper Address?
Humanoid "loco-manipulation"—tasks requiring both walking and hand-eye coordination—is incredibly difficult for current AI models. While Vision-Language-Action (VLA) models are great at individual skills (like "grasp the mug"), they struggle with long-horizon sequences.
The authors identify three main "coupling" problems in monolithic, end-to-end models:
- Temporal Structure: Models often lose track of task progress, leading to "stage confusion" where the robot repeats a step or skips ahead prematurely.
- Precision and Drift: Bipedal robots are inherently unstable. Small movements in the base (pelvis) accumulate, making it nearly impossible for a model to maintain the millimeter-level precision needed to press a small button or place an object accurately.
- Data Imbalance: In long demonstrations, walking takes up most of the time. This biases models toward "conservative" or static behaviors, making the actual manipulation parts of the task harder to learn.
Methods
Instead of trying to force a single neural network to handle everything, PGX decomposes the problem into three distinct layers:
- Timing (Predicate-Gated State Machine): This layer manages when to act. It uses physical "predicates" (sensor-based checks like "is the hand closed and the object lifted?") to gate transitions between stages. It handles navigation and waiting automatically, so the VLA doesn't have to.
- Skills (VLA): The VLA model is only active during manipulation phases. Because it starts from a consistent, "prepared" state provided by the other layers, it can focus entirely on the dynamics of the specific skill.
- Precision (World-Coordinate Locking - WCL): This layer manages where to act. It uses three sub-components to fight drift:
- Object-relative stance: Re-centers the robot's goal position based on the object's current location before every attempt.
- Base station keeping: Actively holds the robot's base still during manipulation.
- World-anchored contact targets: Locks the target (e.g., a button) in the world frame so the arm stays on target even if the robot's body sways.
Datasets
The researchers evaluated PGX using:
- Simulation (Isaac Lab Arena): Three tasks—Box Transport (bimanual), Mug Placement (single-hand), and Coffee Fetching (long-horizon with device interaction). They used 100 MimicGen demonstrations for the box task and 30-50 teleoperated demonstrations for the others.
- Real-World: A coffee fetching task using a Unitree G1 humanoid robot, trained on 44 human demonstrations.
- Model Backbones: They tested the framework with two state-of-the-art VLA models: $\pi_{0.5}$ and GR00T N1.6.
Key Insights and Ideas
- The "Funnel" Effect: By using navigation and coordinate locking to re-establish the starting conditions for every skill, PGX creates a "funnel" that brings the robot back into the specific state where the VLA model was trained to succeed.
- Physical > Semantic: Using hard physical predicates (like distance and contact) for stage transitions is far more robust than relying on a neural network to infer task progress from images alone.
- Decoupling Responsibility: The core insight is that VLAs are best at "how to act," while classical control is better at "when and where to act." Separating these allows the VLA to be more transferable and less prone to the "drifting" errors of a moving humanoid base.

Key Results
The results show that PGX dramatically improves the reliability of existing models without changing their weights:
- Coffee Fetching: On the most complex task, PGX raised the success rate of a fine-tuned $\pi_{0.5}$ model from 0% to 70%. For the GR00T backbone, success jumped from 35% to 80%.
- Robustness to Noise: In "Random Initialization" tests (where the robot starts in a position it never saw during training), PGX maintained high success rates while "flat" end-to-end policies often failed completely.
- Real-World Success: The framework successfully transferred to the Unitree G1 robot, proving that the simulation-to-real gap can be bridged by structured execution.
Limitations and Future Work
- Execution Speed: Because the system waits for physical predicates to be satisfied and performs "station keeping" corrections, it is slower than a purely end-to-end policy.
- Predicate Engineering: Currently, the physical predicates (like "is the mug grasped?") must be manually defined for each task. Future work could involve learning these predicates or using VLMs to generate them.
- Skill Variety: The authors plan to extend the framework to handle a much wider variety of more complex, dynamic skills.
Summary updated