Visual Haptic Reasoning: Estimating Contact Forces by Observing Deformable Object Interactions

Yufei Wang, David Held, Zackory Erickson

Introduction

In the field of robotic caregiving—encompassing tasks such as assistive dressing, bathing, and hygiene—the ability to sense and respond to physical contact is a fundamental requirement for safety and efficacy. While force-torque sensors and tactile skins are effective for rigid-body interactions, "haptic reasoning" with deformable objects like cloth remains a significant challenge. Cloth possesses high-dimensional, non-linear dynamics and creates hundreds of shifting contact points over large surface areas, making it impractical to embed physical sensors throughout the material.

Researchers at Carnegie Mellon University have addressed this by developing a framework for visual haptic reasoning. Their approach enables a robot to infer the location and magnitude of forces applied by cloth onto other objects using only visual data from depth cameras. By training models in physics simulations, they have enabled robots to "see" the touch during complex manipulation tasks.

Methodology: From Pixels to Physics

The core of the proposed system is a data-driven pipeline that translates 3D point clouds into force distributions. The researchers utilized two primary neural network architectures to process this geometric data:

  1. Graph Neural Networks (GNNs): By treating cloth and object points as nodes in a graph, the GNN models how force propagates through the cloth's structure to the contact interface. This architecture mimics the underlying physics of tension and compression within the fabric.
  2. PointNet++: This architecture is used to process 3D point clouds by capturing local geometric features, allowing the model to understand the spatial relationship between the cloth and the object it is touching.

The Prediction Pipeline

The process begins with a depth image, which is converted into a point cloud and segmented into "cloth" and "object" points. To ensure the model is robust enough to transfer from simulation to the real world (sim-to-real), the researchers apply a voxel grid filter to standardize point density.

The model then performs a two-stage prediction:

  • Contact Model: A binary classifier predicts whether a specific point on the object is currently in contact with the cloth.
  • Force Model: A regression model estimates the magnitude (in Newtons) of the normal force at those identified contact points.

A key technical insight is the use of the "Contact Mask." By using the binary classifier to mask the regression model, the researchers significantly reduced prediction noise—preventing the model from predicting tiny, erroneous forces across the entire object surface.

Training and Simulation

Because real-world haptic data for cloth is difficult to collect at scale, the models were trained entirely within the NVIDIA FleX physics simulator. The researchers generated a massive dataset across four diverse tasks:

  • Assistive Dressing: Pulling a hospital gown onto a human arm using 100 different SMPL-X human body meshes.
  • Assistive Bathing: Wiping a washcloth along a human limb.
  • Dish Washing: Cleaning plates of various geometries sourced from the ShapeNet dataset.
  • Primitive Shapes: A generalization task involving pulling a towel over various combinations of cubes, spheres, and cylinders.

This variety ensured that the models learned the general physics of cloth interaction rather than simply memorizing specific trajectories or object shapes.

Key Results and Generalization

The models demonstrated high precision in simulation, achieving a Mean Absolute Error (MAE) as low as 0.188N for dressing tasks. This level of accuracy is significant given that the average forces in these interactions typically range several Newtons.

Furthermore, the study highlighted the power of task-agnostic training. Models trained on dressing, bathing, and dishwashing simultaneously performed nearly as well as those specialized for a single task. This suggests that the networks are successfully learning the fundamental physical principles of how cloth deforms and exerts force.

Real-World Validation

To test the sim-to-real transition, the team deployed the model on a Hello Robot Stretch RE-1 mobile manipulator. In one experiment, the robot cleaned a plate covered in blue powder. The model’s predicted force distribution (and the resulting "cleaned" area) closely matched the actual physical results. This validation proves that focusing on point cloud geometry and relative distances—rather than raw RGB pixels—allows for effective transfer from a "perfect" simulation to the noisy real world.

Limitations and Future Work

While promising, the current framework has several limitations that provide avenues for future research:

  • Occlusion: Cloth naturally obscures the objects it touches. Currently, the system uses a "pre-occlusion" scan of the object, assuming it remains static. Future iterations may incorporate 3D shape completion or capacitive sensing to "see through" the fabric.
  • Dynamic Objects: The current model assumes the object being touched (e.g., a human arm) is stationary. Future work must account for human movement in response to the robot's actions.
  • Force Vectors: The study focuses on normal forces (pushing). Estimating shear forces (friction and rubbing) remains a more complex challenge for future visual-haptic models.

Conclusion

By combining GNNs with high-fidelity physics simulations, this research demonstrates that robots can learn to perceive physical interactions that are otherwise invisible to traditional sensors. This visual haptic reasoning is a crucial step toward safer, more capable robotic assistants in healthcare and domestic environments.

Summary updated

BibTeX
@article{wang2022visual,
  author = {Yufei Wang and David Held and Zackory Erickson},
  title = {{Visual Haptic Reasoning: Estimating Contact Forces by Observing Deformable Object Interactions}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  doi = {10.1109/lra.2022.3199684},
}
Figure