Inferring Interaction Force from Visual Information without Using Physical Force Sensors

Wonjun Hwang, Soo-Chul Lim

TL;DR

This paper introduces a novel deep learning approach that estimates interaction forces using only visual information from sequential images, eliminating the need for physical force sensors. By leveraging Recurrent Neural Networks (RNNs), specifically Long Short-Term Memory (LSTM) networks, the method learns to infer forces from subtle visual cues like object deformation, paving the way for more cost-effective and versatile robotic interaction.

What Problem Does This Paper Address?

Robots interacting with the physical world, especially in tasks requiring dexterity and human-level manipulation, need to sense physical properties like force. Traditionally, this has relied on expensive and sometimes cumbersome physical sensors such as tactile or force/torque sensors. These sensors can be difficult to integrate into certain robotic systems (e.g., surgical robots due to space constraints and sterilization requirements) and can suffer from wear and tear. The challenge is to achieve accurate force estimation without these physical sensors, enabling robots to "feel" their environment through other means, much like humans do. This paper aims to bridge this gap by using only visual input.

Methods

The core of the proposed method is a deep learning architecture based on Recurrent Neural Networks (RNNs), specifically Long Short-Term Memory (LSTM) networks. The authors chose LSTMs because of their ability to model complex temporal dynamics in sequential data, which is crucial for analyzing the subtle changes in object shape over time as a force is applied.

The process involves:

  1. Image Preprocessing: Input images are cropped to focus on the interaction region, normalized to a fixed resolution, and converted to grayscale. Histogram equalization is used to compensate for illumination variations.
  2. LSTM Network: A two-layer LSTM network is employed. The first layer processes sequential image data (40 time steps) to capture temporal dependencies. The output of this layer is then fed into a second LSTM layer for further refinement.
  3. Fully Connected Layers: The outputs from the LSTM layers are then passed through two stacked fully connected (FC) layers. These layers are designed to merge the temporal information and produce a single, static output representing the estimated interaction force. A dropout layer is included between the FC layers to prevent overfitting.
  4. Loss Function: The model is trained using the Mean Squared Error (MSE) between the predicted force and the ground truth force measured by a load cell.

The key novelty lies in using only sequential 2D images and an end-to-end deep learning model to directly map visual inputs to interaction force estimations, without relying on 3D models or additional sensor data.

Datasets

The authors created new datasets for training and evaluating their model. These datasets consist of sequential images captured while applying force to various objects, along with corresponding ground truth force measurements from a load cell.

The datasets were collected using a 149 Hz camera and a custom-built interaction force measurement system. The experiments involved:

  • Three primary materials: Sponge, PET bottle, and a human arm.
  • Variations: For the tube object, experiments were conducted under varying illumination conditions and with different poses of the load cell (e.g., a 5-degree angle).

The data was split into training, validation, and test sets using a repeated random sub-sampling validation protocol. Approximately 6000 sequential images were collected for each material.

Key Insights and Ideas

  • Vision as a "Touchless Sensor": The paper demonstrates that visual information, specifically the deformation of an object, can serve as a proxy for touch, allowing robots to infer interaction forces without direct physical contact. This is analogous to how humans use visual cues to anticipate and understand forces.
  • Temporal Dynamics are Crucial: The success of the LSTM network highlights the importance of analyzing the sequence of visual changes. Subtle, gradual deformations over time provide richer information about the applied force than a single static image.
  • Material Properties Matter: The performance of the model varied across different materials, with sponges showing the best results due to their pronounced and easily visible deformations. This suggests that the effectiveness of visual force estimation is highly dependent on the object's material properties and how it deforms.
  • Robustness to Variations: The model showed reasonable robustness to pose variations but was more sensitive to significant changes in illumination, indicating that lighting conditions can impact the clarity of visual deformation cues.

Key Results

The proposed method demonstrated successful estimation of interaction forces across various materials:

  • Sponge: Achieved the highest accuracy, with an average MSE of 0.0032 and an average PSNR of 25.37 dB. The visual changes in sponges are very apparent, leading to good performance.
  • Human Arm: Showed good performance with an average MSE of 0.010 and an average PSNR of 20.59 dB.
  • PET Bottle: Exhibited lower accuracy (average MSE of 0.031, average PSNR of 15.37 dB), attributed to self-occlusion and less distinct visual deformation cues compared to the sponge.
  • Tube (under variations): The model was robust to a 5-degree pose variation (average MSE of 0.019) but showed weaker performance under significant light changes (average MSE of 0.037).

In general, the predicted forces were reported to be very similar to those measured by the ground truth load cell, with differences often below human perception thresholds for force changes.

Limitations and Future Work

  • Illumination Sensitivity: The model's performance degraded under significant changes in illumination, suggesting that more advanced image processing or a more robust network architecture might be needed to handle extreme lighting variations.
  • Material Dependence: The accuracy is highly dependent on the object's material and its ability to deform visibly. Objects that are very rigid or do not deform significantly might pose a challenge.
  • Self-Occlusion: For objects like the PET bottle, self-occlusion can obscure deformation cues, impacting accuracy.
  • Future Work: The authors plan to develop a three-axis force estimation model that can infer forces in multiple directions, still relying solely on vision sensors. They also aim to integrate this vision-based force estimation into a robot's hand for more practical applications.

Summary updated

Abstract

In this paper, we present an interaction force estimation method that uses visual information rather than that of a force sensor. Specifically, we propose a novel deep learning-based method utilizing only sequential images for estimating the interaction force against a target object, where the shape of the object is changed by an external force. The force applied to the target can be estimated by means of the visual shape changes. However, the shape differences in the images are not very clear. To address this problem, we formulate a recurrent neural network-based deep model with fully-connected layers, which models complex temporal dynamics from the visual representations. Extensive evaluations show that the proposed learning models successfully estimate the interaction forces using only the corresponding sequential images, in particular in the case of three objects made of different materials, a sponge, a PET bottle, a human arm, and a tube. The forces predicted by the proposed method are very similar to those measured by force sensors.
BibTeX
@inproceedings{hwang2017inferring,
  author = {Wonjun Hwang and Soo-Chul Lim},
  title = {{Inferring Interaction Force from Visual Information without Using Physical Force Sensors}},
  booktitle = {Sensors},
  year = {2017},
  url = {https://openreview.net/forum?id=NHIVfUSiG2p3},
}
Figure