Robotic Manipulation with Reinforcement Learning, State Representation Learning, and Imitation Learning (Student Abstract)

Hanxiao ChenAAAI 2021 · DOI

Abstract

This research explores enhancing robotic manipulation efficiency by comparing three learning paradigms: pure Reinforcement Learning (RL), State Representation Learning (SRL) combined with RL, and Imitation Learning. Testing across tasks from simple grasping to complex multi-fingered manipulation reveals that while pure RL suffices for basic tasks, representation or imitation learning is vital for overcoming sample inefficiency and high-dimensional complexity.

Introduction

Robotic manipulation—the ability to interact with and move objects—remains a significant challenge in robotics. Unlike humans, robots learning via standard Reinforcement Learning (RL) face two primary hurdles: sample inefficiency and instability. RL often requires millions of trials for simple tasks and can fail with minor environmental changes. Furthermore, learning from high-dimensional pixel data is computationally expensive. This paper investigates how different learning strategies can mitigate these issues across various simulated manipulation tasks.

Methodology

Three distinct learning approaches were evaluated:

  1. Pure Reinforcement Learning: Standard algorithms like Deep Q-Learning (DQN) and Proximal Policy Optimization (PPO) were applied to basic grasping tasks using simple 1-2 finger grippers.
  2. SRL + RL Pipeline: For goal-oriented tasks like button-pushing, a two-stage process was implemented. State Representation Learning (SRL) techniques (e.g., Auto-Encoders, VAEs, and Robotic Priors) first compressed raw image data into a compact "state space." RL algorithms (DQN, PPO, SAC) were then trained on these compressed states.
  3. Imitation Learning: To address the complexity of multi-fingered dexterous hands, Demo-Augmented Policy Gradient (DAPG) was used. This method bootstraps learning with human demonstrations, providing a guided starting point rather than relying solely on trial-and-error.

Experimental Setup

The research utilized several simulated environments built in PyBullet and MuJoCo, covering a range of task complexities.

Overview of simulation environments used for grasping, button-pushing, and dexterous manipulation.

Environments included:

  • KukaGymEnv-v0 & KukaDiverseObjectEnv-v0: Grasping single or multiple objects.
  • KukaButtonGymEnv-v0 & KukaRandomGymEnv-v0: Custom button-pushing tasks with varying scales and positions.
  • Dexterous Hand Benchmarks: MuJoCo tasks such as object relocation, pen rotation, tool usage, and door-opening.

Discussion: Key Insights

  • Decoupling Perception from Action: The SRL + RL approach demonstrates that a robot's difficulty often stems from visual scene understanding rather than just movement. Extracting features via self-supervised learning first makes subsequent RL training more stable and efficient.
  • Leveraging Robotic Priors: Incorporating physical laws or constraints during the SRL phase helps the robot learn "disentangled" representations that generalize better to new scenarios.
  • Imitation for High-Dimensional Control: For multi-fingered hands with high degrees of freedom, the search space for pure RL is prohibitive. Imitation learning serves as a crucial guide, enabling flexible manipulation where standard RL fails.

Results

  • Grasping Success: DQN and PPO agents successfully solved basic grasping. In the "Diverse Object" environment, agents achieved average reward scores of approximately 50-51.
  • Efficiency Gains: The SRL + RL pipeline effectively addressed sample inefficiency, allowing the robot to learn button-pushing control policies within a limited budget of 50,000 training timesteps.
  • Dexterous Manipulation: The DAPG approach enabled a multi-fingered hand to perform complex tasks like rotating a pen and opening doors, tasks typically difficult for traditional controllers.
Example of complex in-hand manipulation task (pen rotation) solved using Imitation Learning.

Conclusion and Future Work

While effective in simulation, there is significant potential for improvement. Future work could incorporate advanced 3D computer vision techniques to enhance training for dexterous manipulation where spatial orientation is critical. Additionally, exploring new SRL techniques for complex, multi-stage missions remains a key area for research.

Summary updated

BibTeX
@inproceedings{chen2021robotic,
  author = {Hanxiao Chen},
  title = {{Robotic Manipulation with Reinforcement Learning, State Representation Learning, and Imitation Learning (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17881},
}
Figure