Manipulation by Feel: Touch-Based Control with Deep Predictive Models

Stephen Tian, Frederik Ebert, Dinesh Jayaraman, Mayur Mudigonda, Chelsea Finn, Roberto Calandra, Sergey Levine

TL;DR

Researchers from UC Berkeley and Facebook AI Research have developed a framework called Deep Tactile MPC that allows robots to perform precise manipulation tasks using only high-resolution touch sensing. By training a deep predictive model on raw tactile images from a GelSight sensor, the robot learns to "feel" its way through tasks like rolling a die or positioning a ball, outperforming traditional hand-engineered baselines without requiring manual supervision or explicit physics models.

What Problem Does This Paper Address?

While vision is the primary sense for most robots, it has a major flaw in manipulation: occlusion. The moment a robotic finger touches an object, it often blocks its own view of the contact point. Furthermore, vision is "remote"—it doesn't directly capture the friction, pressure, or subtle slips that are critical for dexterity.

Tactile sensing should be the solution, but it faces three hurdles:

  1. Hardware: Most tactile sensors provide only sparse, low-resolution data (like a few pressure points).
  2. Modeling: The physics of contact and friction are incredibly difficult to model analytically.
  3. Goal Specification: It is non-intuitive to define a mathematical "goal" for a tactile sensor (e.g., "what does a successfully held matchstick feel like?").

This paper tackles these by treating tactile sensing as a high-resolution "video" stream and using data-driven deep learning to bypass the need for complex physics equations.

Methods

The authors combine high-resolution hardware with a predictive control loop:

  • GelSight Sensor: They use a custom-built sensor that uses a camera to record the deformations of a soft elastomer (gel) pad. This turns touch into a high-resolution image (48x64 pixels) that captures the geometry of the contact.
  • Deep Predictive Model: Instead of coding physics, they train a deep recurrent convolutional neural network. Given the current tactile image and a sequence of planned movements, the model predicts what the future tactile images will look like.
  • Tactile MPC (Model Predictive Control): At test time, the user provides a "goal tactile image" (a snapshot of what the sensor should see when the task is done). The robot uses the Cross-Entropy Method (CEM) to sample thousands of possible action sequences, predicts their outcomes using the neural network, and executes the first step of the sequence that most closely matches the goal image. This process repeats in a closed loop.

Datasets

The authors collected three distinct datasets through unsupervised autonomous interaction (the robot pokes and moves objects randomly to see what happens):

  • Ball Rolling: 7,400 trajectories of a ball bearing.
  • Analog Stick: 3,000 trajectories of a joystick.
  • 20-Sided Die: 4,500 trajectories of a d20.

Each trajectory consists of 15–18 time steps. No manual labeling was required for training; the robot simply learned the relationship between its actions and the resulting tactile "video."

Key Insights and Ideas

  • Touch as Video: By treating tactile data as a video stream, the researchers could leverage powerful architectures from the computer vision community (like video prediction models) for robotic control.
  • Self-Supervised Goals: The "aha moment" here is that you don't need to define coordinates or forces. You can specify a goal simply by showing the robot a picture of what the contact should look like.
  • Learning "Physics" Implicitly: The model learns complex behaviors—like the fact that a ball rolls in the opposite direction of the finger's movement—purely from raw data, without any knowledge of geometry or mechanics.
  • Handling Discontinuity: For the analog stick task, the robot learned it sometimes needs to break contact (lift the finger) and re-position it to achieve a goal, a complex behavior for traditional controllers.

Key Results

The Deep Tactile MPC was evaluated against a "Centroid Baseline" (a hand-engineered controller that tries to move the center of pressure to a target).

Task Metric Tactile MPC (Ours) Baseline
Ball Rolling Median L2 Error 2.10 mm 2.97 mm
Analog Stick Median L2 Error 5.31 mm 8.86 mm
Die Rolling Success Rate 86.6% 46.6%

Qualitatively, the robot demonstrated the ability to roll a 20-sided die to a specific face and precisely position a ball bearing, even when the object was completely hidden from external cameras.

Limitations and Future Work

  • Short Horizons: The current model is optimized for "servoing" (short-term adjustments). It cannot yet plan long-term, multi-stage strategies (like picking up a tool and then using it).
  • Single Finger: The experiments were limited to a single finger on a 3-axis CNC-style rig. Real-world dexterity requires multi-fingered hands.
  • Simple Rearrangement: The tasks are currently limited to repositioning objects. Future work aims to extend this to assembly, tool use, and more complex in-hand manipulation.
  • Computation: Running MPC with deep models is computationally expensive, currently limiting the control rate to about 1 Hz.

Summary updated

Abstract

Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging. General purpose control techniques that are able to effectively leverage tactile sensing as well as accurate physics models of contacts and forces remain largely elusive, and it is unclear how to even specify a desired behavior in terms of tactile percepts. In this paper, we take a step towards addressing these issues by combining high-resolution tactile sensing with data-driven modeling using deep neural network dynamics models. We propose deep tactile MPC, a framework for learning to perform tactile servoing from raw tactile sensor inputs, without manual supervision. We show that this method enables a robot equipped with a GelSight-style tactile sensor to manipulate a ball, analog stick, and 20-sided die, learning from unsupervised autonomous interaction and then using the learned tactile predictive model to reposition each object to user-specified configurations, indicated by a goal tactile reading. Videos, visualizations and the code are available here: https://sites.google.com/view/deeptactilempc
BibTeX
@inproceedings{tian2019manipulation,
  author = {Stephen Tian and Frederik Ebert and Dinesh Jayaraman and Mayur Mudigonda and Chelsea Finn and Roberto Calandra and Sergey Levine},
  title = {{Manipulation by Feel: Touch-Based Control with Deep Predictive Models}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  doi = {10.1109/icra.2019.8794219},
}

Loading related content...

Figure