Monocular Reconstruction of Neural Tactile Fields

Pavan Mantripragada, Siddhanth Deshmukh, Eadom Dessalene, Manas Desai, Yiannis Aloimonos

TL;DR

Researchers have introduced Neural Tactile Fields (NTFs), a novel 3D representation that allows robots to predict the physical resistance of an object from a single RGB image. By "hallucinating" tactile responses, robots can now distinguish between a soft plant they can push through and a rigid obstacle they must avoid, leading to a massive 85.8% improvement in volumetric 3D reconstruction over current state-of-the-art methods.

What Problem Does This Paper Address?

Most robotic mapping systems treat the world as a collection of static, "impassable" geometries. In a standard occupancy grid, a delicate fern and a concrete pillar look the same: they are both "occupied space."

However, the real world is deformable and reconfigurable. To navigate effectively, a robot needs to know more than just where an object is; it needs to know how that object will respond to force. Is it heavy? Is it compliant? Will it move if I nudge it? Inferring these physical properties from vision alone—especially from a single viewpoint—is a major challenge in robotics that prevents efficient navigation through cluttered, "soft" environments like gardens or warehouses.

Methods

The authors propose a pipeline that transforms a single monocular RGB image into a continuous 3D field of expected tactile pressure.

  1. The Architecture: They build upon the Large Reconstruction Model (LRM), a state-of-the-art architecture that uses a transformer-based "vision backbone" and a "triplane decoder."
  2. The "Tactile" Twist: Instead of just predicting 3D shape (density), they finetune the model to jointly predict geometry and tactile values. They repurpose the density output of the network, scaling it to represent the normalized pressure a tactile sensor would feel at that specific 3D coordinate.
  3. Loss Functions: To train this, they use a combination of:
  4. Photometric Loss: Ensuring the predicted 3D model looks like the input image.
  5. Weighted MSE & IoU Loss: These supervise the "tactile" part of the field, specifically handling the "class imbalance" where most of the space around an object has zero tactile resistance.

Datasets

Because no dataset existed that paired 3D geometry with dense tactile ground truth, the authors built their own:

  • The Dataset: 40 unique objects (20 plants and 20 household items) spanning various weights, stiffnesses, and materials.
  • The Ground Truth: For each object, they used a GelSight Mini (a high-resolution vision-based tactile sensor) to physically poke the object at thousands of points. These measurements were then registered into a 3D volume created via NeRF (Neural Radiance Fields).
  • Benchmark: They compared their results against LRM and Direct3D, two leading methods for single-view 3D reconstruction.

Key Insights and Ideas

  • Physics Improves Geometry: One of the most striking findings is that training a model to predict tactile response actually makes it better at predicting shape. By learning the "cost" of interaction, the model develops a more grounded understanding of the object's boundaries.
  • Tactile Hallucination: The robot doesn't need to touch the object at runtime. It uses the "memory" of physical interactions from its training set to infer the compliance of a new, unseen object just by looking at it.
  • Interaction-Aware Planning: By treating the tactile field as a "cost map," path planners (like RRT*) can find shorter paths. Instead of taking a long detour around a bush, the robot realizes the leaves are low-resistance and chooses to "brush through" them to reach its goal faster.

Key Results

  • Reconstruction Accuracy: The proposed method improved volumetric 3D reconstruction by 85.8% and surface reconstruction (Chamfer Distance) by 26.7% compared to the LRM baseline.
  • F1 Score: At a distance threshold of 0.05, the model achieved an F1 score of 0.2474, nearly four times higher than LRM (0.0622).
  • Planning Efficiency: In simulations, the "interaction-aware" planner successfully utilized compliant regions to reduce path length while maintaining high success rates, whereas geometry-only baselines were forced into long, conservative detours.

Limitations and Future Work

While a major step forward, the authors note several areas for growth:

  • Scale and Diversity: The current dataset is limited to 40 objects on a single mounting surface. Expanding this to more complex, outdoor, or larger-scale scenes is necessary for general-purpose robotics.
  • Directionality: The current NTF is a scalar field (it predicts the magnitude of resistance). However, many objects (like a brush or a hinged door) resist force differently depending on the direction of the push. The authors suggest moving toward Neural Tensor Fields to capture this anisotropy.
  • Friction and Dynamics: The model currently focuses on pressure/compliance but doesn't explicitly model surface friction or complex dynamics (like an object sliding vs. tipping).

Summary updated

Abstract

Robots operating in the real world must plan through environments that deform, yield, and reconfigure under contact, requiring interaction-aware 3D representations that extend beyond static geometric occupancy. To address this, we introduce neural tactile fields, a novel 3D representation that maps spatial locations to the expected tactile response upon contact. Our model predicts these neural tactile fields from a single monocular RGB image -- the first method to do so. When integrated with off-the-shelf path planners, neural tactile fields enable robots to generate paths that avoid high-resistance objects while deliberately routing through low-resistance regions (e.g. foliage), rather than treating all occupied space as equally impassable. Empirically, our learning framework improves volumetric 3D reconstruction by $85.8\%$ and surface reconstruction by $26.7\%$ compared to state-of-the-art monocular 3D reconstruction methods (LRM and Direct3D).
BibTeX
@misc{mantripragada2026monocular,
  author = {Pavan Mantripragada and Siddhanth Deshmukh and Eadom Dessalene and Manas Desai and Yiannis Aloimonos},
  title = {{Monocular Reconstruction of Neural Tactile Fields}},
  year = {2026},
  eprint = {2602.12508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.12508},
}
Figure