Guided Policy Search with Delayed Senor Measurements

Connor Schenck, Dieter FoxarXiv 2016 · arXiv

Abstract

Standard reinforcement learning algorithms often struggle in real-world robotics due to non-Markovian dynamics introduced by sensor delays. This paper presents an extension to Guided Policy Search (GPS) that addresses these challenges through state augmentation and improved dynamics modeling. The approach is demonstrated on a complex robotic pouring task, where a Baxter robot learns to pour precise amounts of liquid despite significant delays in scale measurements, eventually achieving human-level accuracy.

Introduction: The Challenge of Delayed Feedback

In many robotic applications, the assumption of a Markov Decision Process (MDP)—where the next state depends solely on the current state and action—is violated by sensor latency. In the task of robotic pouring, feedback from a digital scale is often delayed by up to a full second due to internal filtering and the time required for liquid to fall.

The experimental setup featuring a Baxter robot performing the pouring task into a bowl on a digital scale.

The authors address this by teaching a Rethink Robotics Baxter robot to pour exactly 100 grams of water. This task is particularly difficult because fluid dynamics are highly non-linear and the sensor readings are significantly delayed by the hardware.

Methodology

The core of the approach is an adaptation of Guided Policy Search (GPS). GPS typically alternates between local trajectory optimization and training a global neural network policy. To handle the specific constraints of the pouring task, three key modifications were implemented:

  • State Augmentation: To restore the Markov property, the state space was augmented with a short history of the previous four states and controls. This allows the policy to account for the 0.5 to 1.0-second delay in the scale readings without requiring complex recurrent architectures.
  • Gaussian Mixture Model (GMM) Priors: Standard GPS uses linear dynamics models. However, the abrupt changes in fluid flow are poorly captured by linear approximations. By using a GMM prior, the system can better approximate the non-linear transitions inherent in pouring.
  • Tight Policy-Optimizer Interleaving: The trajectory optimizer (iLQG) is constrained to stay close to the current neural network policy. This ensures that the optimized trajectories remain within the representational capacity of the global policy, preventing the optimizer from finding paths the network cannot learn.

Experimental Results

The robot was trained using real-world rollouts with varying initial water volumes (between 200g and 400g). The system demonstrated robust convergence, typically stabilizing after 25 iterations.

Error convergence over 50 iterations, showing (a) individual trajectories and (b) mean error with standard deviation.

As shown in the results, the robot achieved a high level of consistency. By iteration 33, the mean error dropped to within 10 grams of the target, with a standard deviation also falling below 10g. This performance is comparable to human accuracy for the same task, demonstrating that simple history stacking combined with flexible dynamics modeling can effectively overcome significant sensor delays in physical systems.

Conclusion

This work demonstrates that complex, non-linear manipulation tasks can be learned directly on physical hardware even in the presence of significant sensor lag. By augmenting the state space and utilizing more expressive dynamics priors, Guided Policy Search becomes a powerful tool for real-world robotic control. Future work aims to integrate these skills into multi-step tasks and incorporate richer sensory inputs like computer vision to foster a better understanding of the physical environment.

Summary updated

Abstract

Guided policy search is a method for reinforcement learning that trains a general policy for accomplishing a given task by guiding the learning of the policy with multiple guiding distributions. Guided policy search relies on learning an underlying dynamical model of the environment and then, at each iteration of the algorithm, using that model to gradually improve the policy. This model, though, often makes the assumption that the environment dynamics are markovian, e.g., depend only on the current state and control signal. In this paper we apply guided policy search to a problem with non-markovian dynamics. Specifically, we apply it to the problem of pouring a precise amount of liquid from a cup into a bowl, where many of the sensor measurements experience non-trivial amounts of delay. We show that, with relatively simple state augmentation, guided policy search can be extended to non-markovian dynamical systems, where the non-markovianess is caused by delayed sensor readings.
BibTeX
@misc{schenck2016guided,
  author = {Connor Schenck and Dieter Fox},
  title = {{Guided Policy Search with Delayed Senor Measurements}},
  year = {2016},
  eprint = {1609.03076},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1609.03076},
}
Figure