Abstract
Standard reinforcement learning algorithms often struggle in real-world robotics due to non-Markovian dynamics introduced by sensor delays. This paper presents an extension to Guided Policy Search (GPS) that addresses these challenges through state augmentation and improved dynamics modeling. The approach is demonstrated on a complex robotic pouring task, where a Baxter robot learns to pour precise amounts of liquid despite significant delays in scale measurements, eventually achieving human-level accuracy.
Introduction: The Challenge of Delayed Feedback
In many robotic applications, the assumption of a Markov Decision Process (MDP)—where the next state depends solely on the current state and action—is violated by sensor latency. In the task of robotic pouring, feedback from a digital scale is often delayed by up to a full second due to internal filtering and the time required for liquid to fall.
The authors address this by teaching a Rethink Robotics Baxter robot to pour exactly 100 grams of water. This task is particularly difficult because fluid dynamics are highly non-linear and the sensor readings are significantly delayed by the hardware.
Methodology
The core of the approach is an adaptation of Guided Policy Search (GPS). GPS typically alternates between local trajectory optimization and training a global neural network policy. To handle the specific constraints of the pouring task, three key modifications were implemented:
- State Augmentation: To restore the Markov property, the state space was augmented with a short history of the previous four states and controls. This allows the policy to account for the 0.5 to 1.0-second delay in the scale readings without requiring complex recurrent architectures.
- Gaussian Mixture Model (GMM) Priors: Standard GPS uses linear dynamics models. However, the abrupt changes in fluid flow are poorly captured by linear approximations. By using a GMM prior, the system can better approximate the non-linear transitions inherent in pouring.
- Tight Policy-Optimizer Interleaving: The trajectory optimizer (iLQG) is constrained to stay close to the current neural network policy. This ensures that the optimized trajectories remain within the representational capacity of the global policy, preventing the optimizer from finding paths the network cannot learn.
Experimental Results
The robot was trained using real-world rollouts with varying initial water volumes (between 200g and 400g). The system demonstrated robust convergence, typically stabilizing after 25 iterations.
As shown in the results, the robot achieved a high level of consistency. By iteration 33, the mean error dropped to within 10 grams of the target, with a standard deviation also falling below 10g. This performance is comparable to human accuracy for the same task, demonstrating that simple history stacking combined with flexible dynamics modeling can effectively overcome significant sensor delays in physical systems.
Conclusion
This work demonstrates that complex, non-linear manipulation tasks can be learned directly on physical hardware even in the presence of significant sensor lag. By augmenting the state space and utilizing more expressive dynamics priors, Guided Policy Search becomes a powerful tool for real-world robotic control. Future work aims to integrate these skills into multi-step tasks and incorporate richer sensory inputs like computer vision to foster a better understanding of the physical environment.
Summary updated