Introduction
Non-prehensile manipulation—interacting with objects without grasping them (e.g., pushing, rolling, or sliding)—is a fundamental skill for versatile robots. However, it is notoriously difficult because the physics of friction and "hybrid dynamics" (switching between sliding and sticking) are hard to model. Most existing solutions rely on perfect vision from external camera setups. In the real world, occlusions are ubiquitous: the robot's own arm might block the view, or a human might step in the way. This paper introduces a robust framework that maintains precise control during prolonged visual occlusions by leveraging tactile feedback and uncertainty-aware estimation.
Methods
The authors propose a multi-stage learning pipeline to bridge the gap between simulation and reality. First, a "teacher" reinforcement learning (RL) agent is trained in simulation with access to perfect ground-truth object poses. To make the system robust, they collect a massive dataset of 750,000 trajectories from teacher checkpoints at various stages of training, capturing both expert behavior and common mistakes.
A Bayesian State Estimator (an LSTM) is trained to predict the object's pose based on noisy vision and force sensors. Crucially, it models two types of uncertainty: aleatoric (inherent data noise) and epistemic (model knowledge gaps). Finally, an uncertainty-aware control policy ($\pi_{est}$) is trained to receive both the estimated pose and the calculated uncertainty, allowing the robot to "know when it doesn't know" and adjust its behavior accordingly.
Experimental Setup
The study primarily uses a custom-built simulation environment in NVIDIA Isaac Sim for training and large-scale testing. Real-world validation was conducted on a KUKA iiwa robot arm equipped with a wrist force/torque sensor and a simple onboard camera. The system was tested for zero-shot sim-to-real transfer, meaning it was deployed on the physical robot without any fine-tuning on real-world data.
Key Results and Insights
The proposed system achieved a 94% success rate in simulation under frequent occlusions, significantly outperforming end-to-end RL (12%) and behavior cloning (38%). In real-world tests, the robot successfully pushed objects to targets even when researchers manually blocked the camera, achieving a 10/10 success rate with 5-second occlusions.
Several key insights emerged from the study:
- Uncertainty as a Feature: By feeding uncertainty into the controller, the robot learns to be more cautious or perform "information-gathering" actions when unsure. For example, the policy learned to deliberately switch contact faces to provide a "tactile spike" of information that clarifies the object's orientation.
- Robustness through Suboptimal Data: Training the estimator on data from "learning" policies rather than just "perfect" ones made it much more robust at recovering from the types of errors a real robot actually makes.
- Estimator Accuracy: Modeling both types of uncertainty reduced the translational error of the state estimator from ~7.6mm (standard MSE) to 4.4mm.
Conclusion
This work demonstrates that combining visuotactile sensing with Bayesian uncertainty modeling enables robust non-prehensile manipulation in occluded environments. While currently focused on fixed-size cuboids, future work aims to generalize this framework to arbitrary, complex shapes and more dynamic tasks like tossing or pivoting.
Summary updated