PourIt!: Weakly-supervised Liquid Perception from a Single Image for Visual Closed-Loop Robotic Pouring

Haitao Lin, Yanwei Fu, Xiangyang Xue

Abstract

Robotic pouring of liquids remains a significant challenge due to the non-rigid, transparent, and refractive nature of liquids, which often leads to sensor failure in standard depth cameras. We present PourIt!, a framework that enables high-precision closed-loop robotic pouring using only a single RGB-D image and weak image-level labels. By leveraging a Transformer-based perception network with a novel feature contrast strategy and a geometric 3D modeling approach, PourIt! achieves robust liquid perception without the need for labor-intensive pixel-wise annotations.

Introduction

Robotic manipulation of liquids is deceptively complex. Unlike rigid objects, liquids lack a fixed shape and exhibit optical properties like transparency and refraction that often confound standard depth sensors. This creates three primary hurdles: the "Annotation Gap," where pixel-level labeling is prohibitively expensive; "Sensor Failure," where infrared-based depth cameras return incomplete data for transparent streams; and the need for "Dynamic Feedback" to handle moving targets.

Figure 1: Overview of the PourIt! framework, highlighting the transition from open-loop bottle-to-bottle distance to closed-loop liquid-to-bottle distance modeling.

Methodology

The PourIt! framework addresses these challenges through a weakly-supervised pipeline that transforms a complex segmentation task into a simpler classification problem.

Weakly-Supervised Perception

The system utilizes a Transformer-based perception network trained on image-level labels (presence or absence of liquid). During inference, the network generates a Class Activation Map (CAM) to localize the liquid stream. To overcome the typical "blurriness" of CAMs, the authors introduced a feature contrast loss. This strategy forces the network to cluster "liquid" features in the latent space while repelling "background" features, resulting in a significantly sharper and more accurate segmentation mask.

Figure 3: The PourIt! architecture, featuring the perception network with Transformer blocks, the contrastive training stage, and the inference pipeline for 3D liquid modeling.

3D Liquid Modeling

To compensate for missing depth data, PourIt! employs a geometric workaround. By estimating the 6-DoF pose of the source container, the system assumes the liquid stream aligns with the bottle's neck and follows the vector of gravity. 2D pixels from the segmented mask are back-projected onto this calculated 3D plane, allowing for the reconstruction of a 3D point cloud of the liquid stream in real-time.

Experimental Setup and Datasets

The framework was evaluated using two primary datasets:

  1. PourIt! Dataset: A new benchmark of 4,064 RGB images featuring diverse liquids (water, coffee, milk, etc.) and backgrounds, including both "seen" and "unseen" test scenarios.
  2. Liquid Dataset: A secondary dataset of 648 trials used to compare performance against fully supervised "oracle" methods.

Results

PourIt! demonstrates state-of-the-art performance in both perception and robotic execution:

  • Perception: The method achieved a 65.9% mIoU on the PourIt! dataset, a 10% improvement over previous weakly-supervised methods like AFA.
  • Robotic Success: In physical trials with a Franka robot, the system achieved a 93.3% success rate in static scenes. Crucially, in dynamic scenes with moving targets, it maintained a success rate of 60-73%, whereas open-loop baselines dropped as low as 33%.
  • Efficiency: The entire perception-to-control loop operates at 10Hz, enabling reactive, closed-loop pouring.

Conclusion and Future Work

PourIt! successfully bridges the gap between weak supervision and high-precision robotic control. While the current model relies on a gravitational assumption that may not hold for highly viscous liquids or high-momentum pours, it provides a robust foundation for liquid manipulation. Future work will explore more complex 3D liquid shapes and extend the framework to tasks such as stirring and cleaning.

Summary updated

Abstract

Liquid perception is critical for robotic pouring tasks. It usually requires the robust visual detection of flowing liquid. However, while recent works have shown promising results in liquid perception, they typically require labeled data for model training, a process that is both time-consuming and reliant on human labor. To this end, this paper proposes a simple yet effective framework PourIt!, to serve as a tool for robotic pouring tasks. We design a simple data collection pipeline that only needs image-level labels to reduce the reliance on tedious pixel-wise annotations. Then, a binary classification model is trained to generate Class Activation Map (CAM) that focuses on the visual difference between these two kinds of collected data, i.e., the existence of liquid drop or not. We also devise a feature contrast strategy to improve the quality of the CAM, thus entirely and tightly covering the actual liquid regions. Then, the container pose is further utilized to facilitate the 3D point cloud recovery of the detected liquid region. Finally, the liquid-to-container distance is calculated for visual closed-loop control of the physical robot. To validate the effectiveness of our proposed method, we also contribute a novel dataset for our task and name it PourIt! dataset. Extensive results on this dataset and physical Franka robot have shown the utility and effectiveness of our method in the robotic pouring tasks. Our dataset, code and pre-trained models will be available on the project page.
BibTeX
@inproceedings{lin2023pourit,
  author = {Haitao Lin and Yanwei Fu and Xiangyang Xue},
  title = {{PourIt!: Weakly-supervised Liquid Perception from a Single Image for Visual Closed-Loop Robotic Pouring}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00029},
}
Figure