Tracking Object State Under In Hand Slippage

36 concepts35 papers1 proposal

In traditional robotics, manipulation planning and state estimation often rely on a convenient simplifying assumption: once a robot's gripper closes around an object, the spatial relationship between the hand and the object remains perfectly rigid. This "rigid grasp assumption" allows algorithms to treat the grasped object as a predictable, static extension of the robot's own kinematic chain.

However, physical interaction is inherently messy. To sense, move, or use an object, a robot must apply forces to it. The paradox of contact is that these very forces frequently cause the object to slip, rotate, or shift within the gripper. When this happens, the static coordinate transformations used by the robot's controller instantly become inaccurate, leading to cascading errors or outright task failure.

To build truly robust manipulation systems, researchers are moving away from static assumptions and developing frameworks that explicitly expect, track, and compensate for contact-induced displacement.

The Paradox of Contact-Induced Displacement

The fundamental challenge of tactile perception is that gathering information about an object can inadvertently alter its state. As Kai-yuh Hsiao et al. observe in Task-Driven Tactile Exploration, physical contact during exploration introduces a frustrating trade-off: the act of touching an object to resolve pose uncertainty can apply enough force to displace or rotate it, introducing entirely new uncertainty.

This problem is not unique to tactile probing. Any interaction—whether it is an in-hand handover or an acoustic tap—can induce slippage:

How Slippage Corrupts Control and Kinematics

When a robot uses a tool or manipulates an articulated object, the consequences of unmodeled slippage extend from bad perception to control failure.

For instance, force estimation frameworks often map the deformations of a compliant gripper directly to external forces acting on a tool. However, as H. J. Terry Suh et al. point out in SEED: Series Elastic End Effectors in 6D for Visuotactile Tool Use, if the tool slips within the grasp, the underlying geometric model becomes invalid, corrupting the force calculations.

[Static Model Assumption]  --> Gripper Moves --> Object Slips --> Kinematic Model Corrupted
[Dynamic Tracking Model]  --> Gripper Moves --> Slip Tracked  --> Coordinate Frame Updated

This vulnerability is particularly acute when interacting with articulated objects like doors or levers. Adrian Röfer et al. demonstrate in Online Estimation of Articulated Objects with Factor Graphs using Vision and Proprioceptive Sensing that if a gripper slips on a handle, a purely kinematic estimation system will incorrectly interpret that slippage as object articulation. Because the system assumes a rigid connection, the end-effector's unintended displacement corrupts the calculated joint parameters of the object.

Ultimately, for high-precision operations like industrial insertion, Dhruv Metha Ramesh et al. show in Integrating Model-based Control and RL for Sim2Real Transfer of Tight Insertion Policies that failing to compensate for post-grasp shifts invalidates coordinate transformations, causing the policy to miss sub-millimeter targets.

Continuous Tracking via Visuotactile Fusion

To overcome the limitations of the rigid grasp assumption, modern frameworks treat the in-hand pose of an object as a dynamic variable that must be continuously estimated. One of the most effective ways to achieve this is by fusing global visual data with local tactile feedback.

Vision provides a broad, unoccluded view of the scene, while tactile sensors capture high-resolution local physics—such as shear forces and micro-slips—at the contact interface. Jayjun Lee et al. leverage this in ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation by combining visual point clouds with tactile "shear fields." This allows a robot to track an object's shifting pose and its external contact points simultaneously, even when the object slides significantly within the hand.

Similarly, Byron Boots et al. introduce Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing, which uses temporal tactile data to map complex contact patterns without assuming a rigid grasp. By continuously updating the spatial mapping between tactile signals and object geometry, the system remains accurate despite significant in-hand slippage.

When dealing with compliant or deformable objects, where the object's shape changes alongside its position, Youngsun Wi et al. present VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects. By learning a latent representation of contact dynamics, their framework prevents coordinate frame misalignment when flexible tools twist or slide within the gripper.

Active Perception and Post-Grasp Re-Perception

Beyond continuous tracking, robots can handle grasp instability by actively planning for it. Instead of executing a trajectory blindly, a robot can perform "post-grasp re-perception" or alter its movements to evaluate how an object has shifted.

In kPAM-SC: Generalizable Manipulation Planning using KeyPoint Affordance and Shape Completion, Wei Gao et al. counter post-grasp slippage by integrating a continuous state estimation loop that updates the object's pose mid-trajectory, ensuring that the downstream manipulation plan remains valid.

[Initial Grasp] 
       │
       ▼
[Physical Movement] ──> Forces Applied ──> Unintended Slip Occurs
       │
       ▼
[Active Re-Perception] ──> Quantify Uncertainty ──> Update Pose Model ──> Task Success

This active approach is highly effective for identifying intrinsic object properties, such as the center of mass (CoM). When a robot lifts and reorients an arbitrary object, gravity exerts torques that can cause the item to shift within the fingers. Shengmiao Jin et al. address this in Learning to Double Guess: An Active Perception Approach for Estimating the Center of Mass of Arbitrary Objects. Their framework uses a Bayesian Neural Network to quantify pose uncertainty and actively selects an informative second "guess" or movement. By expecting and modeling the physical shift during reorientation, the system prevents the force-torque estimation errors that typically plague static models.

Finally, to prevent slips from causing catastrophic drops, the authors of Combined Physics and Event Camera Simulator for Slip Detection focus on the temporal resolution of the slip itself. By training neural networks on high-frequency simulated event camera data, they enable robots to detect the onset of subtle micro-slips in real time, allowing controllers to adjust grip force before the object breaks free entirely.

By abandoning the assumption of a rigid grasp and treating the contact interface as a dynamic, evolving system, modern robotic frameworks achieve the flexibility required to operate in unpredictable, real-world environments.

Key Foundational Work

3D Object Reconstruction from Hand-Object Interactions (2015) — contributed 1 concept elements to this direction.
Grasp pose estimation in human-robot manipulation tasks using wearable motion sensors (2015) — contributed 1 concept elements to this direction.
Task-Driven Tactile Exploration (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_rigid_grasp_assumption_0dcee88d,
  title = {rigid grasp assumption},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/rigid-grasp-assumption/rdr_0dcee88d524c410e9bc48ae53c859ab9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_rigid_grasp_assumption_0dcee88d_v1,
  title = {rigid grasp assumption},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/rigid-grasp-assumption/rdr_0dcee88d524c410e9bc48ae53c859ab9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake