FingerSLAM: Closed-loop Unknown Object Localization and Reconstruction from Visuo-tactile Feedback

Jialiang Zhao, Maria Bauzá, Edward H. Adelson

Abstract

In this paper, we address the problem of using visuo-tactile feedback for 6-DoF localization and 3D reconstruction of unknown in-hand objects. We propose FingerSLAM, a closed-loop factor graph-based pose estimator that combines local tactile sensing at finger-tip and global vision sensing from a wrist-mount camera. FingerSLAM is constructed with two constituent pose estimators: a multi-pass refined tactile-based pose estimator that captures movements from detailed local textures, and a single-pass vision-based pose estimator that predicts from a global view of the object. We also design a loop closure mechanism that actively matches current vision and tactile images to previously stored key-frames to reduce accumulated error. FingerSLAM incorporates the two sensing modalities of tactile and vision, as well as the loop closure mechanism with a factor graph-based optimization framework. Such a framework produces an optimized pose estimation solution that is more accurate than the standalone estimators. The estimated poses are then used to reconstruct the shape of the unknown object incrementally by stitching the local point clouds recovered from tactile images. We train our system on real-world data collected with 20 objects. We demonstrate reliable visuo-tactile pose estimation and shape reconstruction through quantitative and qualitative real-world evaluations on 6 objects that are unseen during training.

Introduction

When a robot picks up an unknown object, it faces a fundamental "chicken and egg" problem: determining the object's shape requires knowing its pose relative to the hand, but tracking the pose accurately during manipulation requires an understanding of the object's geometry. Standard computer vision often struggles in these scenarios due to hand occlusions or challenging material properties, such as transparency. While tactile sensing provides high-resolution local detail, it is prone to "drift"—small estimation errors that accumulate over time and distort the final map. FingerSLAM addresses this by implementing Simultaneous Localization and Mapping (SLAM) for in-hand manipulation, providing a robust, drift-resistant framework for real-time tracking and 3D reconstruction.

Methods

The FingerSLAM system utilizes a Factor Graph framework to fuse disparate sensory estimates into an optimized pose solution. The system consists of three primary components:

  1. Visuo-Tactile Odometry:
    • Tactile Branch: A GelSight Wedge sensor generates high-resolution depth maps of the contact patch. A neural network-based multi-step refinement process iteratively "warps" previous tactile images to match current ones, enhancing accuracy for minute movements.
  2. Vision Branch: A wrist-mounted RGB camera (RealSense D405) provides a global 6-DoF pose estimate. This "big picture" context prevents the tactile sensor from losing global orientation.
  3. Loop Closure: To mitigate accumulated drift, the system stores key-frames of visual and tactile data. When the robot re-encounters a previously visited area, a loop closure factor is added to the graph, effectively "snapping" the estimated trajectory back to the correct pose.
  4. Incremental Reconstruction: Optimized poses are used to stitch local tactile depth patches into a global 3D point cloud, allowing the robot to build a model of the unknown object as it explores it.
The hardware setup and data collection pipeline, showing the integration of GelSight tactile sensing and RGB vision.

Experimental Setup and Datasets

The system was trained on a custom dataset of 20 3D-printed objects from the YCB and ABC datasets, totaling approximately 40,000 tactile and 25,000 vision samples after data augmentation. Evaluation was conducted on 6 unseen objects, including complex 3D-printed shapes and real-world glass jars. Ground truth for training and validation was provided by an OptiTrack motion capture system.

A high-resolution tactile depth map produced by the GelSight sensor, capturing the local geometry of the object.

Results and Discussion

FingerSLAM achieved a median pose tracking accuracy of 2.8mm and 2.4 degrees across all test objects. Key findings include:

  • Complementary Modalities: Tactile sensing proved superior for small movements (<5mm), while vision provided necessary stability for larger movements (>10mm).
  • Handling Transparency: The system successfully reconstructed a completely transparent glass jar. While vision-only systems often fail on clear materials due to reflections, FingerSLAM's reliance on tactile data for local tracking allowed it to maintain high accuracy.
  • Impact of Loop Closure: Without loop closure, 3D reconstructions eventually became misaligned due to drift. The active matching of current contacts to stored key-frames was essential for recognizable shape reconstruction.

Conclusion and Future Work

FingerSLAM demonstrates that fusing local tactile resolution with global visual context within a closed-loop factor graph significantly improves in-hand object tracking and mapping. Current limitations include the assumption of object rigidity and the requirement for constant contact. Future work will explore incorporating force vectors for richer movement data and moving toward probabilistic inference to allow the factor graph to dynamically weigh sensor reliability based on uncertainty.

Summary updated

Abstract

In this paper, we address the problem of using visuo-tactile feedback for 6-DoF localization and 3D reconstruction of unknown in-hand objects. We propose FingerSLAM, a closed-loop factor graph-based pose estimator that combines local tactile sensing at finger-tip and global vision sensing from a wrist-mount camera. FingerSLAM is constructed with two constituent pose estimators: a multi-pass refined tactile-based pose estimator that captures movements from detailed local textures, and a single-pass vision-based pose estimator that predicts from a global view of the object. We also design a loop closure mechanism that actively matches current vision and tactile images to previously stored key-frames to reduce accumulated error. FingerSLAM incorporates the two sensing modalities of tactile and vision, as well as the loop closure mechanism with a factor graph-based optimization framework. Such a framework produces an optimized pose estimation solution that is more accurate than the standalone estimators. The estimated poses are then used to reconstruct the shape of the unknown object incrementally by stitching the local point clouds recovered from tactile images. We train our system on real-world data collected with 20 objects. We demonstrate reliable visuo-tactile pose estimation and shape reconstruction through quantitative and qualitative real-world evaluations on 6 objects that are unseen during training.
BibTeX
@inproceedings{zhao2023fingerslam,
  author = {Jialiang Zhao and Maria Bauzá and Edward H. Adelson},
  title = {{FingerSLAM: Closed-loop Unknown Object Localization and Reconstruction from Visuo-tactile Feedback}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=89UUPYqIot},
}
Figure