Abstract
In this letter, we investigate learning forward dynamics models and multi-step prediction of state variables (long-term prediction) for contact-rich manipulation. The problems are formulated in the context of model-based reinforcement learning (MBRL). We focus on two aspects—discontinuous dynamics and data-efficiency—both of which are important in the identified scope and pose significant challenges to State-of-the-Art methods. We contribute to closing this gap by proposing a method that explicitly adopts a specific hybrid structure for the model while leveraging the uncertainty representation and data-efficiency of Gaussian process. Our experiments on an illustrative moving block task and a 7-DOF robot demonstrate a clear advantage when compared to popular baselines in low data regimes.
The Challenge of Contact-Rich Manipulation
In robotics, "contact-rich" tasks—where a robot frequently touches, slides against, or impacts objects—are notoriously difficult to model. These interactions create discontinuous dynamics: a sudden impact causes an abrupt jump in velocity that standard smooth mathematical models cannot easily capture.
Furthermore, most modern approaches, such as those used in Model-Based Reinforcement Learning (MBRL), require massive amounts of data. In real-world scenarios, running thousands of trials to learn a model is impractical due to physical wear and tear on the robot. The challenge lies in creating a model that is both data-efficient and capable of long-term prediction (forecasting many steps into the future) despite these sharp "jumps" in system behavior.
Methodology: A Hybrid Probabilistic Framework
The proposed method treats the robot's environment as a Hybrid Dynamical System. Instead of one monolithic model, the task is decomposed into discrete "modes" (e.g., free motion, sliding, or stuck).
- Mode Discovery: A Dirichlet Process Gaussian Mixture Model (DPGMM) is used to automatically cluster trial data into different physical modes without human labeling.
- Mode Dynamics: Each mode is modeled using a Gaussian Process (GP). GPs are chosen for their extreme data efficiency and ability to provide a measure of uncertainty.
- Guard Functions: A Multi-class Support Vector Machine (SVM) acts as a "guard," predicting when the system is likely to switch from one mode to another based on the current state and action.
- Reset Maps: This is a critical addition. When a transition occurs (like hitting a wall), a specific GP "reset map" calculates the discontinuous jump in state, such as a sudden loss of velocity.
- Uncertainty Propagation: To predict far into the future, the framework uses the Unscented Transform (UT) and Monte Carlo sampling to pass probability distributions through these non-linear models.
Experimental Evaluation
The researchers evaluated their approach on two main setups:
- Simulated 1D Sliding Mass: A block subjected to frictionless motion, sticking, and slipping, serving as a controlled environment to test handling of abrupt velocity jumps.
- 7-DOF YuMi Robot Experiment: Real-world data from an ABB YuMi robot performing a peg-in-hole style task involving free motion and sliding along a steel edge.
To specifically measure data efficiency, the models were tested in both "small" (15 trials) and "large" (40 trials) data regimes.
Key Results and Insights
- The Power of Reset Maps: Most learning models try to "smooth over" discontinuities. By explicitly using reset maps, this model acknowledges that impacts are instantaneous jumps, leading to much more realistic simulations of contact.
- Probabilistic Switching: Because the exact moment of contact is uncertain, the model allows for "multimodal" predictions, tracking multiple possible outcomes (e.g., contact vs. free motion) simultaneously.
- Superior Accuracy and Efficiency: In the YuMi robot task, the proposed method achieved an RMSE of 0.15, significantly outperforming uncertainty-aware neural networks (uANN) which scored 0.47. Furthermore, the hybrid model performed better with just 15 trials than the baselines did with 40, proving its utility for rapid real-world deployment.
Conclusion and Future Work
By imposing a hybrid automata structure as a "strong prior," the authors enable robots to learn complex contact physics from a handful of examples. While highly effective in low-data regimes, future work will need to address the cubic scaling of Gaussian Processes for larger datasets and the challenge of adapting to previously unseen physical modes.
Summary updated