Manipulation by Feel: Touch-Based Control with Deep Predictive Models

Stephen Tian, Frederik Ebert, Dinesh Jayaraman, Mayur Mudigonda, Chelsea Finn, Roberto Calandra, Sergey Levine

Abstract

Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging. General purpose control techniques that are able to effectively leverage tactile sensing as well as accurate physics models of contacts and forces remain largely elusive, and it is unclear how to even specify a desired behavior in terms of tactile percepts. In this paper, we take a step towards addressing these issues by combining high-resolution tactile sensing with data-driven modeling using deep neural network dynamics models. We propose deep tactile MPC, a framework for learning to perform tactile servoing from raw tactile sensor inputs, without manual supervision. We show that this method enables a robot equipped with a GelSight-style tactile sensor to manipulate a ball, analog stick, and 20-sided die, learning from unsupervised autonomous interaction and then using the learned tactile predictive model to reposition each object to user-specified configurations, indicated by a goal tactile reading.

Introduction

While vision is the primary sense for most robots, it has a major flaw in manipulation: occlusion. The moment a robotic finger touches an object, it often blocks its own view of the contact point. Furthermore, vision is "remote"—it doesn't directly capture the friction, pressure, or subtle slips that are critical for dexterity.

Tactile sensing should be the solution, but it faces significant hurdles. Most tactile sensors provide only sparse, low-resolution data, and the physics of contact and friction are incredibly difficult to model analytically. Moreover, it is non-intuitive to define a mathematical "goal" for a tactile sensor. This paper introduces Deep Tactile MPC, a framework that treats tactile sensing as a high-resolution "video" stream and uses data-driven deep learning to bypass the need for complex physics equations.

Methods

The authors combine high-resolution hardware with a predictive control loop. The core of the system is the GelSight Sensor, a custom-built device that uses a camera to record the deformations of a soft elastomer (gel) pad. This turns touch into a high-resolution image (48x64 pixels) that captures the geometry of the contact.

Comparison of human finger touch and the high-resolution GelSight tactile sensor.

Instead of hand-coding physics, a Deep Predictive Model—specifically a deep recurrent convolutional neural network—is trained. Given the current tactile image and a sequence of planned movements, the model predicts future tactile images. At test time, the robot uses Tactile MPC (Model Predictive Control). A user provides a "goal tactile image," and the robot uses the Cross-Entropy Method (CEM) to sample thousands of action sequences, executing the one that most closely matches the goal.

Experimental Setup and Datasets

The framework was evaluated on three distinct tasks, with data collected through unsupervised autonomous interaction. The robot "pokes" and moves objects randomly to learn the relationship between its actions and the resulting tactile "video."

Experimental setups for the three manipulation tasks: ball rolling, analog stick positioning, and die rolling.

The datasets included:

  • Ball Rolling: 7,400 trajectories of a ball bearing.
  • Analog Stick: 3,000 trajectories of a joystick.
  • 20-Sided Die: 4,500 trajectories of a d20.

By treating touch as video, the model learns complex behaviors—such as the fact that a ball rolls in the opposite direction of the finger's movement—purely from raw data. It even learns when it must break contact to reposition, a difficult behavior for traditional controllers.

Key Results

Deep Tactile MPC was evaluated against a "Centroid Baseline" (a hand-engineered controller). The results demonstrate significant improvements in precision and success rates.

Task Metric Tactile MPC (Ours) Baseline
Ball Rolling Median L2 Error 2.10 mm 2.97 mm
Analog Stick Median L2 Error 5.31 mm 8.86 mm
Die Rolling Success Rate 86.6% 46.6%

Qualitatively, the robot demonstrated the ability to roll a 20-sided die to a specific face and precisely position a ball bearing, even when the object was completely hidden from external cameras.

Discussion and Future Work

While successful, the current model is optimized for short-term "servoing" and cannot yet plan long-term, multi-stage strategies. Additionally, the experiments were limited to a single finger on a 3-axis rig; real-world dexterity will require multi-fingered hands. Future work aims to extend this to assembly, tool use, and more complex in-hand manipulation, while also addressing the computational expense of running MPC with deep models.

Summary updated

Abstract

Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging. General purpose control techniques that are able to effectively leverage tactile sensing as well as accurate physics models of contacts and forces remain largely elusive, and it is unclear how to even specify a desired behavior in terms of tactile percepts. In this paper, we take a step towards addressing these issues by combining high-resolution tactile sensing with data-driven modeling using deep neural network dynamics models. We propose deep tactile MPC, a framework for learning to perform tactile servoing from raw tactile sensor inputs, without manual supervision. We show that this method enables a robot equipped with a GelSight-style tactile sensor to manipulate a ball, analog stick, and 20-sided die, learning from unsupervised autonomous interaction and then using the learned tactile predictive model to reposition each object to user-specified configurations, indicated by a goal tactile reading. Videos, visualizations and the code are available here: https://sites.google.com/view/deeptactilempc
BibTeX
@inproceedings{tian2019manipulation,
  author = {Stephen Tian and Frederik Ebert and Dinesh Jayaraman and Mayur Mudigonda and Chelsea Finn and Roberto Calandra and Sergey Levine},
  title = {{Manipulation by Feel: Touch-Based Control with Deep Predictive Models}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  doi = {10.1109/icra.2019.8794219},
}
Figure