FingerSLAM: Closed-loop Unknown Object Localization and Reconstruction from Visuo-tactile Feedback

Jialiang Zhao, Maria Bauzá, Edward H. Adelson

TL;DR

FingerSLAM is a new "closed-loop" system that allows a robot to figure out exactly how it’s holding an unknown object and what that object looks like, just by touching it and looking at it. By fusing local, high-resolution tactile data from a GelSight sensor with global visual data from a wrist camera into a factor graph, the system achieves precise 6-DoF pose tracking (2.8mm accuracy) and incremental 3D reconstruction, even for challenging items like transparent glassware.

What Problem Does This Paper Address?

When a robot picks up an object it has never seen before, it faces a "chicken and egg" problem: to know the object's shape, it needs to know the object's pose relative to its hand; but to track the pose accurately as the object moves or slips, it needs to understand the object's geometry.

Standard computer vision often fails here because the robot's own hand occludes the object, or the object might be transparent or featureless. Tactile sensing provides incredible local detail but suffers from "drift"—small errors in estimation that accumulate over time, leading to a distorted final map. This paper tackles Simultaneous Localization and Mapping (SLAM) for in-hand manipulation, aiming to provide a robust, drift-resistant way to track and reconstruct unknown objects in real-time.

Methods

The core of FingerSLAM is a Factor Graph, a mathematical framework used to fuse different (and sometimes conflicting) sensory estimates into one "best guess."

  1. Visuo-Tactile Odometry:
    • Tactile Branch: Uses a GelSight Wedge sensor to produce high-resolution depth maps of the contact patch. The authors implemented a multi-step refinement process (similar to Iterative Closest Point) where a neural network iteratively "warps" the previous tactile image to match the current one, significantly improving accuracy for small movements.
  2. Vision Branch: A wrist-mounted RGB camera uses a MobileNetV3-based architecture to provide a global estimate of the object's 6-DoF pose. While less precise than touch for tiny movements, vision provides the "big picture" context that prevents the tactile sensor from getting "lost."
  3. Loop Closure: This is the "closed-loop" part of the title. The system stores "key-frames" of what it has seen and felt. If the robot touches a spot it has visited before, a loop closure factor is added to the graph. This acts as a "reset" button for accumulated drift, snapping the estimated trajectory back to reality.
  4. Incremental Reconstruction: As the robot explores the object, the optimized poses are used to "stitch" the local tactile depth patches into a global 3D point cloud.

Datasets

  • Training: The authors collected a custom dataset using 20 3D-printed objects from the YCB and ABC datasets. They gathered ~7,000 raw data points and used data augmentation (artificial rotations/translations) to expand this to 40,000 tactile and 25,000 vision samples.
  • Evaluation: The system was tested on 6 unseen objects: 4 3D-printed shapes and 2 real-world glass jars (one painted orange, one completely transparent).
  • Ground Truth: An OptiTrack motion capture system was used during data collection to provide the "true" pose of the objects for training and validation.

Key Insights and Ideas

  • Complementary Strengths: The researchers found that tactile sensing is superior for small movements (<5mm) because of its high resolution, while vision is better for larger movements (>10mm) where tactile patches might not overlap enough to correlate.
  • The Transparency Win: One of the most striking "aha" moments is the performance on the transparent glass jar. Standard vision-only systems usually fail on clear glass due to reflections and lack of features. Because FingerSLAM relies on tactile data for local tracking and uses the factor graph to weigh inputs, it maintained high accuracy where vision-only methods would have crashed.
  • Loop Closure is Essential: Without loop closure, the 3D reconstructions (stitching tactile patches) eventually become "misaligned" due to drift. Actively checking if a new contact looks/feels like an old one is what makes the reconstruction recognizable.

Key Results

  • Tracking Accuracy: FingerSLAM achieved a median pose tracking accuracy of 2.8mm and 2.4 degrees across all test objects.
  • Ablation Success: The "Combined" model (Vision + Tactile) outperformed standalone estimators. Adding "Loop Closure" (FingerSLAM) further improved results across the board, especially for long sequences.
  • Robustness: The system successfully reconstructed objects with smooth surfaces (like the "Mini Bleach" bottle) and complex textures (like the "Graphic" block), proving it doesn't rely on a specific type of surface feature.

Limitations and Future Work

  • Rigidity Assumption: The current system assumes objects are rigid. It cannot yet handle deformable objects like sponges or rubber ducks.
  • Single Object: The framework is designed for a single object in the scene.
  • Constant Contact: It assumes the object is always in contact with the finger and within the camera's view.
  • Future Directions: The authors suggest incorporating force vectors (sensing the direction of push/pull) to provide even more information about the object's movement. They also propose moving toward probabilistic inference, where the neural networks predict their own uncertainty to help the factor graph decide which sensor to trust more in a given moment.

Summary updated

Abstract

In this paper, we address the problem of using visuo-tactile feedback for 6-DoF localization and 3D reconstruction of unknown in-hand objects. We propose FingerSLAM, a closed-loop factor graph-based pose estimator that combines local tactile sensing at finger-tip and global vision sensing from a wrist-mount camera. FingerSLAM is constructed with two constituent pose estimators: a multi-pass refined tactile-based pose estimator that captures movements from detailed local textures, and a single-pass vision-based pose estimator that predicts from a global view of the object. We also design a loop closure mechanism that actively matches current vision and tactile images to previously stored key-frames to reduce accumulated error. FingerSLAM incorporates the two sensing modalities of tactile and vision, as well as the loop closure mechanism with a factor graph-based optimization framework. Such a framework produces an optimized pose estimation solution that is more accurate than the standalone estimators. The estimated poses are then used to reconstruct the shape of the unknown object incrementally by stitching the local point clouds recovered from tactile images. We train our system on real-world data collected with 20 objects. We demonstrate reliable visuo-tactile pose estimation and shape reconstruction through quantitative and qualitative real-world evaluations on 6 objects that are unseen during training.
BibTeX
@inproceedings{zhao2023fingerslam,
  author = {Jialiang Zhao and Maria Bauzá and Edward H. Adelson},
  title = {{FingerSLAM: Closed-loop Unknown Object Localization and Reconstruction from Visuo-tactile Feedback}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=89UUPYqIot},
}

Loading related content...

Figure