Reaching optimally over the workspace: A machine learning approach

Didier Marin, Olivier Sigaud2012 · DOI

TL;DR

This paper proposes a machine learning approach to create fast, reactive controllers for robotic arm movements that mimic optimal control strategies. By learning from demonstrations and then refining with policy search, they achieve near-optimal performance 20,000 times faster than traditional methods, enabling real-time control for complex tasks.

What Problem Does This Paper Address?

Human motor control is remarkably adept at coordinating movements to achieve goals efficiently. Theories suggest this relies on Optimal Control (OC) principles, where the system optimizes a trade-off between effort and reward. However, applying these OC methods directly to robots is computationally prohibitive for real-time applications due to the need for constant replanning. This paper addresses the challenge of creating robotic controllers that can achieve near-optimal performance reactively, mimicking human capabilities without the computational burden of traditional OC.

Methods

The authors propose a two-stage machine learning approach:

  1. Learning from Demonstration (LfD) with XCSF: They first use a system called the Near-Optimal Planning System (NOPS), which is based on OC principles but computationally expensive, to generate "demonstration" trajectories. These trajectories are then used to train a classifier system called XCSF (eXternal Classifier System). XCSF is a rule-based system that can approximate functions and learn from data. It clusters the state space and learns local linear models to predict the appropriate action for a given state. This results in a reactive policy, termed the "XCSF policy" ($\pi_\theta$).
  2. Policy Improvement with Cross-Entropy Policy Search (CEPS): To further enhance the performance of the XCSF policy, they employ a direct policy search method called CEPS. CEPS is an iterative algorithm that starts with a distribution of policies and progressively refines it by sampling policies, evaluating their performance (e.g., through simulated robot trajectories), and updating the distribution towards better-performing policies. This method is robust and can improve the policy's performance, especially in areas where the initial LfD data was sparse or suboptimal.

Datasets

The paper does not explicitly mention or introduce new datasets. Instead, it uses simulated robot arm trajectories generated by their NOPS system as the "demonstration data" for training the XCSF policy. The evaluation is performed on simulated two-degrees-of-freedom planar arms.

Key Insights and Ideas

  • Bridging the Gap: The core idea is to leverage the strengths of both OC (for defining optimality) and machine learning (for real-time execution and generalization). NOPS provides the "what" (optimal behavior), and XCSF/CEPS provide the "how" (fast, reactive execution).
  • LfD as a Foundation: Learning from demonstrations generated by a computationally expensive but theoretically sound method (NOPS) provides a strong initial policy that can be further refined.
  • XCSF for Reactivity and Generalization: XCSF's ability to approximate functions and its rule-based nature allow it to create a reactive controller that can generalize to states not explicitly seen in the demonstrations.
  • CEPS for Fine-Tuning: CEPS offers a robust way to optimize the learned policy, particularly improving performance in challenging regions of the workspace or for specific task parameters.
  • Emergent Movement Time: The paper demonstrates that the learned controller can reproduce the emergent property of movement time, which arises from a trade-off between reward and effort, as predicted by OC models.

Key Results

  • Speedup: The learned controller ($\pi_\theta$) is approximately 20,000 times faster than the original NOPS method, executing trajectories in milliseconds compared to minutes.
  • Performance: The XCSF policy achieves performance close to the NOPS, especially for targets near the demonstration set. CEPS further improves this performance globally across the workspace.
  • Generalization: The learned policy shows good generalization capabilities, both for reaching multiple targets from a single start position and for reaching a single target from multiple start positions. Performance degradation is observed as the test conditions move further from the training data.
  • Movement Time Control: The learned policy qualitatively reproduces the relationship between goal reward and movement time observed with NOPS, demonstrating its ability to capture this emergent property.

Limitations and Future Work

  • Suboptimal Performance After Optimization: The authors acknowledge that even after CEPS optimization, the performance is not yet as close to the OC method as desired, particularly concerning generalization and performance improvement related to reward values.
  • Limited LfD: They cannot perform further LfD after CEPS optimization, suggesting a need for more integrated learning approaches.
  • Future Work:
    • Lifelong Learning: Implementing lifelong learning to continuously optimize controller parameters over time.
  • Advanced Optimization: Exploring more sophisticated optimization alternatives to CEM, such as CMA-ES or PI².
  • Active Learning: Developing active learning capabilities to allow the system to intelligently select trajectories for learning and to choose between LfD and CEPS based on current performance.
  • Application to Assistance Robots: Applying the learned approach to assist robots for tasks like sit-to-stand transfers, adapting in real-time to individual patient needs.
  • Whole-Body Control: Extending the techniques to whole-body control for humanoid robots like iCub to achieve more human-like movements.

Summary updated

Abstract

Recent theories of Human Motor Control explain our outstanding coordination capabilities by calling upon an Optimal Control (OC) framework. But OC methods are generally too expensive to be applied on-line and in realtime as would be required to perform everyday movements. An alternative method consists in obtaining a pre-computed feedback policy that performs optimally while being executed reactively. One way to get such a pre-computed policy consists in tuning a parametrized reactive controller so that it converges to optimal behavior. In this paper, we demonstrate a method to obtain such a reactive controller that (i) adapts the time of movement based on a compromise between the amount of reward and the effort required to get it, (ii) provides an efficient trajectory from any point to any point in the workspace, (iii) learns from demonstrations of optimal trajectories, (iv) is improving its performance over accumulated experience.
BibTeX
@misc{marin2012reaching,
  author = {Didier Marin and Olivier Sigaud},
  title = {{Reaching optimally over the workspace: A machine learning approach}},
  year = {2012},
  doi = {10.1109/biorob.2012.6290743},
}
Figure