In classical robotics, the world is beautifully predictable. If you know the 3D positions of three non-collinear points on an object, you know the exact position and orientation of every single atom in that object. This is the rigid-body assumption, and it forms the mathematical foundation for almost all traditional robotic perception, state estimation, and control. By reducing an object to a simple 6-Degree-of-Freedom (6DoF) coordinate frame, robots can easily plan trajectories, calculate grasp points, and predict collisions.
But the real world is squishy. From the clothes we fold to the food we prepare, many everyday objects are non-rigid. When a robot interacts with fabric, ropes, sponges, or liquids, the rigid-body assumption breaks down entirely. Deformable objects possess an infinite number of degrees of freedom, meaning their shapes change dynamically and unpredictably under gravity and external forces.
To bring robots out of structured factories and into human environments, researchers are actively developing new paradigms that move beyond the rigid-body constraint.
Where Classical Frameworks Break Down
When a robotic system optimized for rigid objects encounters a deformable material, failures cascade across the entire perception-action loop.
Trajectory Generation and Spatial Transformations
Traditional trajectory planners rely heavily on calculating fixed geometric transformations. For example, if a robot needs to rotate a tool, it computes a standard 3D rotation matrix. However, as Tomoya Yoshida et al. point out in Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Vision, assuming object rigidity fails completely for tasks involving sponges or fabrics. Because the object's geometry alters continuously during interaction, a single 6DoF trajectory cannot accurately capture or guide the manipulation.
Grasping Metrics and Contact Mechanics
In rigid-body mechanics, grasp success is often determined by mathematical guarantees like force closure—ensuring that the forces applied by the gripper can counteract any external disturbance. Yet, Isabella Huang et al. demonstrate in DefGraspSim: Physics-Based Simulation of Grasp Outcomes for 3D Deformable Objects that traditional force closure metrics are insufficient for compliant items. When an object deforms under pressure, its contact patches shift and its internal geometry changes, rendering rigid-body mathematical guarantees obsolete.
State Representation and Keypoints
Many modern imitation learning pipelines track objects using a discrete set of 3D keypoints or graph nodes. While this compresses the visual world into a manageable state space, it struggles with topological changes. As Wenlong Huang et al. observe in ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation, forward models that assume objects are rigid over short time horizons suffer from severe inaccuracies when predicting the behaviors of highly deformable materials.
Furthermore, George Jiayuan Gao et al. show in Out-of-Distribution Recovery with Object-Centric Keypoint Inverse Policy For Visuomotor Imitation Learning that keypoint-based error recovery frameworks cannot easily generalize to items like fabric, because a sparse set of rigid points cannot capture the fluid, continuous deformations required to correct a failed action.
Rigid Body Paradigm:
[Object] ───> Compressed to 6DoF Frame ───> Predictable Linear Math ───> Success
Deformable Reality:
[Object] ───> Infinite Degrees of Freedom ───> Shifting Contact Patches ───> Model Breakdown
Simulating the Squishy World
If we cannot represent deformable objects as simple coordinate frames, how do we model them? The answer lies in physics-integrated representations that capture internal material properties like stress, strain, and elasticity.
Instead of relying on rigid approximations, the authors of DefGraspSim utilize GPU-accelerated Finite Element Method (FEM) simulations. FEM discretizes a continuous soft object into a mesh of small, interconnected elements, allowing the system to model how forces propagate through the material. This lets the robot "see" invisible internal stresses and predict exactly how an item will warp before executing a grasp.
Similarly, Siyu Ma et al. introduced GRIP: A General Robotic Incremental Potential Contact Simulation Dataset for Unified Deformable-Rigid Coupled Grasping. By leveraging an Incremental Potential Contact (IPC) simulator, their framework accurately models the complex, non-linear contact mechanics that occur when a soft robotic gripper squishes against a compliant object.
This shift toward physics-aware simulation even allows robots to interact with complex natural environments. For instance, Jayadeep Jacob et al. demonstrate in Learning to Simulate Tree-Branch Dynamics for Manipulation that instead of treating vegetation as a static, rigid obstacle to be avoided, a robot can use simulation-based Bayesian inference to predict branch deformation. This enables the robot to actively push aside foliage to accomplish tasks like fruit picking.
Multimodal Perception and Scalable Learning
Transitioning these physics-aware models to the real world requires a fundamental rethink of robotic perception. Visual data alone is often inadequate because a soft object can change its internal state without changing its immediate visual silhouette, or it may be partially blocked by the robot's own gripper.
Co-Deforming Tactile Sensing
When a robot manipulates a flexible object using a soft gripper, a unique challenge arises: both the sensor and the object deform simultaneously. Miquel Oller et al. tackle this in Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control by explicitly modeling the membrane physics of the tactile sensor itself. By decoupling the sensor's internal structural dynamics from the object's shape changes, the robot can accurately estimate the state of sponges or cables during high-deformation tasks.
To scale this type of learning without requiring millions of physical trials, Shaohong Zhong developed a framework in Touching a NeRF: Leveraging Neural Radiance Fields for Tactile Sensory Data Generation to translate visual Neural Radiance Fields into synthetic tactile feedback. However, as the author notes, fully capturing the complex surface deformations of non-rigid materials remains an open frontier.
Overcoming the Data Gap in VLA Models
Modern Vision-Language-Action (VLA) models show immense promise for general robot control, but they are bottlenecked by data. While we can easily generate synthetic data for rigid objects, doing so for deformable objects is computationally expensive.
As Haoyun Li et al. explain in MimicDreamer: Aligning Human and Robot Demonstrations for Scalable VLA Training, standard generative diffusion processes and inverse kinematics pipelines frequently fail when trying to model the high-degree-of-freedom physics of liquids or cloth. To circumvent this, researchers are building advanced world models that combine visual diffusion with underlying physics constraints, allowing robots to "imagine" and learn the fluid geometry of non-rigid materials directly from human demonstration videos.
By replacing the rigid-body assumption with representations that embrace deformation, material physics, and multimodal feedback, the robotics community is closing the gap between rigid automation and the flexible, compliant reality of human environments.
Key Foundational Work
Vision Based Robot Behavior: Tools and Testbeds for Real-World AI Research (1993) — contributed 1 concept elements to this direction.