On Training Flexible Robots using Deep Reinforcement Learning

Zach Dwiel, Madhavun Candadai, Mariano J. PhielippIROS 2019 · DOI

TL;DR

This paper explores using Deep Reinforcement Learning (DRL) to train flexible robots, demonstrating that DRL can effectively learn robust control policies even with varying degrees of robot flexibility. Surprisingly, adding more complex sensors like IMUs didn't always improve performance, suggesting that careful sensor selection is crucial for successful DRL in flexible robotics.

What Problem Does This Paper Address?

Traditional industrial robots often rely on rigid designs and precise dynamical models for control, which limits their adaptability and makes them unsuitable for many real-world applications. Building accurate dynamical models for flexible robots is notoriously challenging due to their inherent compliance and complex behavior. This paper addresses the need for effective training methodologies for these flexible robots, aiming to achieve robust and adaptable control without relying on intricate pre-built models. The authors investigate whether Deep Reinforcement Learning (DRL), a powerful learning paradigm, can overcome these challenges and enable end-to-end training of flexible robotic systems.

Methods

The core of this research lies in applying Deep Deterministic Policy Gradients (DDPG), a DRL algorithm known for handling continuous action and observation spaces, to train flexible robots. The authors developed a simulated single-link robot arm with a controllable "pseudo-joint" that mimics flexibility. They systematically varied the degree of this flexibility to test the DRL agent's learning capabilities.

Key methodological aspects include:

  • Policy Search with DRL: Utilizing DDPG for end-to-end learning, where the agent learns control policies directly from sensor inputs and reward signals, bypassing the need for explicit dynamical models.
  • Varying Flexibility: The simulated robot's flexibility was controlled by adjusting a spring joint's time constant, allowing for a spectrum of behaviors from rigid to highly compliant.
  • Sensor Configuration Analysis: The study rigorously compared the performance of DRL policies trained with different sensor inputs:
  • Joint angle sensors only.
  • Inertial Measurement Units (IMUs) only.
  • A combination of joint angle sensors and IMUs.
  • Information-Theoretic Analysis: To understand the impact of sensor choices, the researchers measured mutual information between sensor observations and end-effector position/reward under random and trained policies.

Datasets

The paper does not use pre-existing datasets. Instead, it introduces a simulated benchmark environment using the MuJoCo physics engine within the OpenAI Gym framework. This environment features a flexible robot arm with adjustable levels of flexibility, serving as the primary platform for training and evaluation.

Key Insights and Ideas

  • DRL is Capable of Learning Flexible Robot Control: The study confirms that DRL, specifically DDPG, can successfully learn efficient and robust control policies for flexible robots across various tasks and flexibility levels. This is a significant step towards enabling adaptable robotics without complex modeling.
  • Robustness to Flexibility: Learned policies demonstrated remarkable robustness, performing consistently well even when tested on flexibility levels different from those encountered during training. This suggests that DRL can generalize and adapt to variations in hardware dynamics.
  • Sensor Choice is Critical, More Isn't Always Better: Counter-intuitively, adding more complex sensors (IMUs) did not consistently lead to improved performance. In some cases, it even degraded performance, highlighting that the relevance and informativeness of sensor data for the specific task and learning algorithm are more important than sheer quantity.
  • Information vs. Performance: While IMUs provided more information about the robot's dynamics and end-effector position, this increased information did not translate to better learning outcomes. This suggests a complex relationship between information content and the agent's ability to leverage it effectively.

Key Results

  • Consistent Performance Across Flexibility Levels: DDPG was able to learn effective policies for tasks like "Reacher," "Reacher Stay," "Thrower," and "Ant" across a wide range of flexibility levels, often performing as well as or better than on rigid hardware.
  • Robustness Demonstrated: Policies trained at one flexibility level generalized well to other levels, showing minimal performance degradation. For instance, the "Reacher Stay" and "Ant" tasks showed particularly consistent performance across all tested flexibility levels.
  • Joint Angles Suffice: In most tasks, using only joint angle sensors yielded comparable or superior performance to using IMUs or a combination of both. The "Reacher" task showed no difference, while "Thrower" and "Ant" tasks performed worse with IMUs.
  • Information Theory Insights: The mutual information analysis confirmed that IMUs did increase the information about end-effector position and reward. However, this did not directly correlate with improved DRL performance, underscoring the complexity of information utilization in learning.

Limitations and Future Work

  • Sensor Choice Sensitivity: The paper acknowledges that DRL can be sensitive to sensor choice, and the counter-intuitive finding that more sensors don't always help is a key area for further exploration.
  • Limited Task Set: While four tasks were explored, future work could investigate a broader range of tasks to further validate the findings.
  • Simulation vs. Real-World: The experiments were conducted in simulation. Future work should focus on transferring these learned policies to real flexible robots to validate their practical applicability.
  • Algorithm Exploration: The authors suggest exploring other DRL algorithms (e.g., TRPO, PPO, SAC) that might be less susceptible to sensor choice.
  • Understanding Information-Learning Relationship: Further research is needed to systematically understand the relationship between the information content of observations and an agent's learning ability in deep networks.

Summary updated

BibTeX
@inproceedings{dwiel2019training,
  author = {Zach Dwiel and Madhavun Candadai and Mariano J. Phielipp},
  title = {{On Training Flexible Robots using Deep Reinforcement Learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  doi = {10.1109/iros40897.2019.8968516},
}
Figure