In traditional robotics, manipulation planning and state estimation often rely on a convenient simplifying assumption: once a robot's gripper closes around an object, the spatial relationship between the hand and the object remains perfectly rigid. This "rigid grasp assumption" allows algorithms to treat the grasped object as a predictable, static extension of the robot's own kinematic chain.
However, physical interaction is inherently messy. To sense, move, or use an object, a robot must apply forces to it. The paradox of contact is that these very forces frequently cause the object to slip, rotate, or shift within the gripper. When this happens, the static coordinate transformations used by the robot's controller instantly become inaccurate, leading to cascading errors or outright task failure.
To build truly robust manipulation systems, researchers are moving away from static assumptions and developing frameworks that explicitly expect, track, and compensate for contact-induced displacement.
The Paradox of Contact-Induced Displacement
The fundamental challenge of tactile perception is that gathering information about an object can inadvertently alter its state. As Kai-yuh Hsiao et al. observe in Task-Driven Tactile Exploration, physical contact during exploration introduces a frustrating trade-off: the act of touching an object to resolve pose uncertainty can apply enough force to displace or rotate it, introducing entirely new uncertainty.
This problem is not unique to tactile probing. Any interaction—whether it is an in-hand handover or an acoustic tap—can induce slippage:
- Perceptual Misalignment: When a robot attempts to reconstruct an object's 3D shape, assuming a rigid grasp leads to severe geometric distortion if the object shifts. Qi Ye et al. demonstrate in In-Hand 3D Object Reconstruction from a Monocular RGB Video that ignoring in-hand rotation during active manipulation invalidates the fixed relative pose models required for accurate visual tracking. Similarly, Shaoxiong Wang et al. show in 3D Shape Perception from Monocular Vision, Touch, and Shape Priors that if an object shifts during physical probing, the spatial registration between the initial visual estimate and subsequent tactile data is lost.
- Acoustic and Bimanual Failures: Even non-visual modalities suffer. Jiaxun Liu et al. note in SonicSense: Object Perception from In-Hand Acoustic Vibration that object shifting misaligns the mapping between vibration signals and 3D coordinates. In bimanual operations, Tianshuang Qiu et al. find in Omni-Scan: Creating Visually-Accurate Digital Twin Object Models Using a Bimanual Robot with Handover and Gaussian Splat Merging that subtle slips during gripper-to-gripper transfers cause mathematical alignment failures between different scan segments.
How Slippage Corrupts Control and Kinematics
When a robot uses a tool or manipulates an articulated object, the consequences of unmodeled slippage extend from bad perception to control failure.
For instance, force estimation frameworks often map the deformations of a compliant gripper directly to external forces acting on a tool. However, as H. J. Terry Suh et al. point out in SEED: Series Elastic End Effectors in 6D for Visuotactile Tool Use, if the tool slips within the grasp, the underlying geometric model becomes invalid, corrupting the force calculations.
[Static Model Assumption] --> Gripper Moves --> Object Slips --> Kinematic Model Corrupted
[Dynamic Tracking Model] --> Gripper Moves --> Slip Tracked --> Coordinate Frame Updated
This vulnerability is particularly acute when interacting with articulated objects like doors or levers. Adrian Röfer et al. demonstrate in Online Estimation of Articulated Objects with Factor Graphs using Vision and Proprioceptive Sensing that if a gripper slips on a handle, a purely kinematic estimation system will incorrectly interpret that slippage as object articulation. Because the system assumes a rigid connection, the end-effector's unintended displacement corrupts the calculated joint parameters of the object.
Ultimately, for high-precision operations like industrial insertion, Dhruv Metha Ramesh et al. show in Integrating Model-based Control and RL for Sim2Real Transfer of Tight Insertion Policies that failing to compensate for post-grasp shifts invalidates coordinate transformations, causing the policy to miss sub-millimeter targets.
Continuous Tracking via Visuotactile Fusion
To overcome the limitations of the rigid grasp assumption, modern frameworks treat the in-hand pose of an object as a dynamic variable that must be continuously estimated. One of the most effective ways to achieve this is by fusing global visual data with local tactile feedback.
Vision provides a broad, unoccluded view of the scene, while tactile sensors capture high-resolution local physics—such as shear forces and micro-slips—at the contact interface. Jayjun Lee et al. leverage this in ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation by combining visual point clouds with tactile "shear fields." This allows a robot to track an object's shifting pose and its external contact points simultaneously, even when the object slides significantly within the hand.
Similarly, Byron Boots et al. introduce Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing, which uses temporal tactile data to map complex contact patterns without assuming a rigid grasp. By continuously updating the spatial mapping between tactile signals and object geometry, the system remains accurate despite significant in-hand slippage.
When dealing with compliant or deformable objects, where the object's shape changes alongside its position, Youngsun Wi et al. present VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects. By learning a latent representation of contact dynamics, their framework prevents coordinate frame misalignment when flexible tools twist or slide within the gripper.
Active Perception and Post-Grasp Re-Perception
Beyond continuous tracking, robots can handle grasp instability by actively planning for it. Instead of executing a trajectory blindly, a robot can perform "post-grasp re-perception" or alter its movements to evaluate how an object has shifted.
In kPAM-SC: Generalizable Manipulation Planning using KeyPoint Affordance and Shape Completion, Wei Gao et al. counter post-grasp slippage by integrating a continuous state estimation loop that updates the object's pose mid-trajectory, ensuring that the downstream manipulation plan remains valid.
[Initial Grasp]
│
▼
[Physical Movement] ──> Forces Applied ──> Unintended Slip Occurs
│
▼
[Active Re-Perception] ──> Quantify Uncertainty ──> Update Pose Model ──> Task Success
This active approach is highly effective for identifying intrinsic object properties, such as the center of mass (CoM). When a robot lifts and reorients an arbitrary object, gravity exerts torques that can cause the item to shift within the fingers. Shengmiao Jin et al. address this in Learning to Double Guess: An Active Perception Approach for Estimating the Center of Mass of Arbitrary Objects. Their framework uses a Bayesian Neural Network to quantify pose uncertainty and actively selects an informative second "guess" or movement. By expecting and modeling the physical shift during reorientation, the system prevents the force-torque estimation errors that typically plague static models.
Finally, to prevent slips from causing catastrophic drops, the authors of Combined Physics and Event Camera Simulator for Slip Detection focus on the temporal resolution of the slip itself. By training neural networks on high-frequency simulated event camera data, they enable robots to detect the onset of subtle micro-slips in real time, allowing controllers to adjust grip force before the object breaks free entirely.
By abandoning the assumption of a rigid grasp and treating the contact interface as a dynamic, evolving system, modern robotic frameworks achieve the flexibility required to operate in unpredictable, real-world environments.
Key Foundational Work
3D Object Reconstruction from Hand-Object Interactions (2015) — contributed 1 concept elements to this direction.
Grasp pose estimation in human-robot manipulation tasks using wearable motion sensors (2015) — contributed 1 concept elements to this direction.
Task-Driven Tactile Exploration (2010) — contributed 1 concept elements to this direction.