Abstract
Robotic pouring of liquids remains a significant challenge due to the non-rigid, transparent, and refractive nature of liquids, which often leads to sensor failure in standard depth cameras. We present PourIt!, a framework that enables high-precision closed-loop robotic pouring using only a single RGB-D image and weak image-level labels. By leveraging a Transformer-based perception network with a novel feature contrast strategy and a geometric 3D modeling approach, PourIt! achieves robust liquid perception without the need for labor-intensive pixel-wise annotations.
Introduction
Robotic manipulation of liquids is deceptively complex. Unlike rigid objects, liquids lack a fixed shape and exhibit optical properties like transparency and refraction that often confound standard depth sensors. This creates three primary hurdles: the "Annotation Gap," where pixel-level labeling is prohibitively expensive; "Sensor Failure," where infrared-based depth cameras return incomplete data for transparent streams; and the need for "Dynamic Feedback" to handle moving targets.
Methodology
The PourIt! framework addresses these challenges through a weakly-supervised pipeline that transforms a complex segmentation task into a simpler classification problem.
Weakly-Supervised Perception
The system utilizes a Transformer-based perception network trained on image-level labels (presence or absence of liquid). During inference, the network generates a Class Activation Map (CAM) to localize the liquid stream. To overcome the typical "blurriness" of CAMs, the authors introduced a feature contrast loss. This strategy forces the network to cluster "liquid" features in the latent space while repelling "background" features, resulting in a significantly sharper and more accurate segmentation mask.
3D Liquid Modeling
To compensate for missing depth data, PourIt! employs a geometric workaround. By estimating the 6-DoF pose of the source container, the system assumes the liquid stream aligns with the bottle's neck and follows the vector of gravity. 2D pixels from the segmented mask are back-projected onto this calculated 3D plane, allowing for the reconstruction of a 3D point cloud of the liquid stream in real-time.
Experimental Setup and Datasets
The framework was evaluated using two primary datasets:
- PourIt! Dataset: A new benchmark of 4,064 RGB images featuring diverse liquids (water, coffee, milk, etc.) and backgrounds, including both "seen" and "unseen" test scenarios.
- Liquid Dataset: A secondary dataset of 648 trials used to compare performance against fully supervised "oracle" methods.
Results
PourIt! demonstrates state-of-the-art performance in both perception and robotic execution:
- Perception: The method achieved a 65.9% mIoU on the PourIt! dataset, a 10% improvement over previous weakly-supervised methods like AFA.
- Robotic Success: In physical trials with a Franka robot, the system achieved a 93.3% success rate in static scenes. Crucially, in dynamic scenes with moving targets, it maintained a success rate of 60-73%, whereas open-loop baselines dropped as low as 33%.
- Efficiency: The entire perception-to-control loop operates at 10Hz, enabling reactive, closed-loop pouring.
Conclusion and Future Work
PourIt! successfully bridges the gap between weak supervision and high-precision robotic control. While the current model relies on a gravitational assumption that may not hold for highly viscous liquids or high-momentum pours, it provides a robust foundation for liquid manipulation. Future work will explore more complex 3D liquid shapes and extend the framework to tasks such as stirring and cleaning.
Summary updated