A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms

Oliver Kroemer, Scott Niekum, George Dimitri KonidarisarXiv 2019 · OpenReview

Abstract

A key challenge in intelligent robotics is creating robots that are capable of directly interacting with the world around them to achieve their goals. The last decade has seen substantial growth in research on the problem of robot manipulation, which aims to exploit the increasing availability of affordable robot arms and grippers to create robots capable of directly interacting with the world to achieve their goals. Learning will be central to such autonomous systems, as the real world contains too much variation for a robot to expect to have an accurate model of its environment, the objects in it, or the skills required to manipulate them, in advance. We aim to survey a representative subset of that research which uses machine learning for manipulation. We describe a formalization of the robot manipulation learning problem that synthesizes existing research into a single coherent framework and highlight the many remaining research opportunities and challenges.

Introduction: The Complexity of Manipulation

Robot manipulation—the use of actuators to purposefully alter the state of the environment—remains a central goal of robotics. Unlike mobile robots that primarily navigate through space, manipulation robots must contend with a world that is messy, unpredictable, and physically complex. Traditional engineering approaches, which rely on precise pre-programmed models, often fail when faced with the infinite variety of object shapes, materials, and environmental conditions found in the real world.

The field faces three primary technical hurdles:

  1. Underactuation: Robots can control their own joints, but the objects they interact with only move through contact.
  2. Hybrid Dynamics: Manipulation involves distinct "modes." For instance, moving an arm through the air follows continuous physics, but striking a surface or grasping an object introduces sudden, discontinuous changes in dynamics.
  3. High-Dimensionality: Raw sensory data (pixels, point clouds) is massive, requiring the robot to distill relevant features from a sea of noise.
Robot manipulation tasks like cutting involve complex hybrid dynamics and contact-rich interactions.

A Unified Framework: The Task Family

To organize the diverse research in this field, the authors propose formalizing manipulation as a Task Family. Rather than solving a single Markov Decision Process (MDP), manipulation is modeled as a distribution over MDPs. In this framework, the robot's hardware and action space remain constant, while the environment and specific goals vary.

A critical component of this formalization is the Context Vector (τ). This vector represents variables that remain fixed during a specific task—such as the weight of a tool or the friction of a surface—but vary across the broader task family. By learning policies that are functions of both the current state and the context, robots can achieve the level of generalization required for real-world deployment.

The hierarchy of skills and organizing principles for robot manipulation learning.

Technical Pillars of Manipulation Learning

The literature is categorized into five essential pillars:

  • Learning State Spaces: Discovering relevant features like object poses and material properties from high-dimensional sensors.
  • Transition Models: Learning the "physics" of how specific actions alter the environment's state.
  • Skill Policies: Developing motor control strategies through Reinforcement Learning (RL) or Imitation Learning (IL).
  • Characterizing Skills: Identifying "preconditions" (when a skill is applicable) and "effects" (the resulting state change).
  • Hierarchical Structures: Composing primitive skills (e.g., "reach," "grasp") into complex, multi-step behaviors.

Key Insights and Future Directions

The survey highlights several paradigm shifts in the field. Interactive Perception is emphasized as a necessity; robots must "poke" or touch objects to resolve epistemic uncertainty that vision alone cannot address. Furthermore, Object-Centric Representations have proven more effective than learning from raw pixels, as they allow a robot to transfer "opening" logic from a cabinet to a drawer.

The authors also highlight Dynamic Movement Primitives (DMPs) as a highly effective parametric policy for robotics, as they guarantee goal reaching while allowing for flexible, learned motion shapes.

While progress has been significant, several frontiers remain:

  • In Situ Learning: Transitioning from isolated training phases to robots that learn safely while performing tasks.
  • Safety Guarantees: Developing mathematical frameworks to ensure neural network policies do not cause damage.
  • Common-Sense Physics: Enabling robots to understand intuitive physical priors (e.g., liquids leak from cracked containers) without requiring exhaustive datasets.
  • Sample Efficiency: Reducing the massive amount of data currently required to learn even simple manipulation tasks.

By synthesizing these challenges and methods into a coherent framework, this review provides a roadmap for the next generation of autonomous manipulation systems.

Summary updated

Abstract

A key challenge in intelligent robotics is creating robots that are capable of directly interacting with the world around them to achieve their goals. The last decade has seen substantial growth in research on the problem of robot manipulation, which aims to exploit the increasing availability of affordable robot arms and grippers to create robots capable of directly interacting with the world to achieve their goals. Learning will be central to such autonomous systems, as the real world contains too much variation for a robot to expect to have an accurate model of its environment, the objects in it, or the skills required to manipulate them, in advance. We aim to survey a representative subset of that research which uses machine learning for manipulation. We describe a formalization of the robot manipulation learning problem that synthesizes existing research into a single coherent framework and highlight the many remaining research opportunities and challenges.
BibTeX
@misc{kroemer2019review,
  author = {Oliver Kroemer and Scott Niekum and George Dimitri Konidaris},
  title = {{A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms}},
  year = {2019},
  eprint = {1907.03146},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.03146},
}
Figure