Introduction
In robotic manipulation, the success of a grasp is frequently measured by the robot's ability to lift an object vertically. However, functional tasks—such as pouring from a bottle or positioning a tool—require the robot to re-orient the object into various "holding poses." A grasp that appears stable in a vertical orientation may fail when subjected to the gravitational torques and contact force shifts associated with tilting or rotating.
The PoseIt dataset addresses this gap by shifting the regulatory focus from initial pick-and-place success to long-term grasp stability across a wide range of orientations. By integrating high-resolution tactile sensing with vision and force data, this research provides a framework for predicting whether a held object will remain secure during complex arm movements.
The PoseIt Dataset
The PoseIt dataset consists of 1,840 grasp datapoints collected from 26 distinct household objects. Unlike previous datasets that prioritize object variety or initial grasp points, PoseIt focuses on the temporal dynamics of the "holding" phase.
Data Collection Pipeline
The researchers implemented a three-stage experimental protocol to ensure a robust ground truth for stability:
- Grasp: The robot executes a parallel-jaw grasp and lifts the object.
- Pose: The arm moves the object into one of 16 sampled holding poses, categorized by their orientation relative to gravity.
- Shake: To rigorously test the grasp, the robot performs a shaking motion. This phase serves as the definitive test; if the object slips or falls during the shake, the grasp is labeled "unstable."
Multi-Modal Sensing
To capture the nuances of the interaction, the dataset includes:
- Tactile Data: High-resolution imprints from GelSight sensors.
- Visual Data: RGB-D video sequences of the manipulation.
- Kinematic and Force Data: 6-axis force/torque (F/T) values, joint angles, velocities, and gripper force.
Methodology
The core of the stability prediction model is a Long Short-Term Memory (LSTM) network designed to process the temporal sequences from the "Grasp" and "Pose" phases.
Feature Extraction and Architecture
The model utilizes pre-trained ResNet50 backbones to extract spatial features from both RGB images and GelSight tactile imprints. These features are then fed into the LSTM, which analyzes the 20-timestep sequence preceding the "Shake" phase. By learning the dynamics of how an object shifts during the transition to a new pose, the model can predict the outcome of the shake before it occurs.
Addressing Data Imbalance
A significant challenge in stability datasets is the prevalence of "stable" labels, as most successful lifts do not immediately fail. To prevent the model from becoming biased toward the majority class, the authors employed Deferred-Resampling (DRS). This technique allows the model to first learn general features from the full dataset before focusing on the difficult, underrepresented cases where an object appears stable but is actually prone to failure.
Key Results and Analysis
The experimental results underscore the necessity of multi-modal sensing in complex manipulation tasks.
The Power of Multi-Modality
The combined Vision + Tactile model achieved an accuracy of 85.2% on unseen poses, outperforming the vision-only baseline by 13.2% and the tactile-only baseline by 3.4%. While tactile data proved to be a more reliable individual predictor of slip than vision, the synergy between the two modalities provided the most robust results. Vision offers context regarding the object's geometry and orientation, while tactile sensing provides direct feedback on the friction and pressure at the contact interface.
Pose-Dependent Stability Trends
The study revealed that stability is a dynamic property. Interestingly, the number of unstable cases actually decreased by 14% when moving from the initial lift to certain holding poses, suggesting that some orientations naturally stabilize the object within the gripper. However, the subsequent "Shake" phase increased instability by 4%, highlighting the "sneaky" nature of grasps that look secure but cannot withstand external disturbances.
Generalization
The model demonstrated strong generalization to new objects when the holding poses were familiar. However, performance decreased when the robot was tested on entirely novel pose orientations. This suggests that while the model learns the "feel" of a stable grasp, the specific geometry of the pose space remains a critical factor that requires extensive training data.
Conclusion
The PoseIt dataset and the accompanying LSTM-based model represent a significant step toward more reliable robotic manipulation. By demonstrating that stability is pose-dependent and that multi-modal sensing—specifically the inclusion of high-resolution tactile data—is essential for prediction, this work provides a foundation for more "aware" robotic systems.
Future work will look toward prescriptive models: instead of merely predicting failure, robots should be able to determine the optimal sequence of movements to maximize grasp stability throughout a task. Expanding the dataset to include multi-fingered grippers and a wider variety of objects will further enhance the robustness of these predictive systems.
Summary updated