Abstract
Accurate liquid state estimation is a critical requirement for robotic pouring tasks. However, perceiving transparent liquids remains a significant challenge due to their lack of distinct visual features and the failure of traditional depth sensors. We present a novel self-supervised segmentation pipeline that identifies transparent liquids from static RGB images without requiring manual annotations or specialized hardware like thermal cameras. Our approach utilizes a generative model to translate images of colored liquids into synthetic transparent liquid images, leveraging an unpaired dataset for training. By automatically generating labels from colored liquids via background subtraction, we train a segmentation network capable of high-accuracy transparent liquid detection. We demonstrate the efficacy of this method through robotic pouring experiments, achieving precise closed-loop control based on perceived liquid height.
Introduction
In the field of robotics, the ability to handle liquids is essential for applications ranging from domestic assistance to laboratory automation. While solid object manipulation has seen significant progress, liquid perception remains a notorious "hard mode" problem for computer vision. Transparent liquids, such as water, are particularly difficult because they possess no fixed shape and are nearly invisible to standard sensors. They do not reflect light in a predictable manner; instead, they refract the background, creating complex visual distortions that confuse standard depth sensors like LiDAR or infrared.
Existing methods to overcome these challenges often involve environmental constraints, such as using checkerboard backgrounds to detect refraction patterns or heating the liquid to make it visible to thermal cameras. These constraints limit the deployability of robots in "in-the-wild" settings like kitchens or labs. This research addresses the problem by developing a system that can segment transparent liquids using only a single, static RGB image, requiring no manual human labeling or environmental modifications.
Methodology: The Self-Supervised Pipeline
The core of the proposed system is a self-supervised training pipeline that bridges the gap between easily observable colored liquids and difficult-to-see transparent ones. The process is divided into four primary stages:
1. Unpaired Data Collection
The researchers collected two distinct datasets: one featuring containers filled with colored liquid (green water) and another with transparent liquid. A key advantage of this method is that the datasets do not need to be "paired." The containers do not need to be in the same position, nor do they need to contain the same volume of liquid across the two sets, making data collection significantly easier.
2. Image-to-Image Translation
To create training data for transparent liquids, the team employed Contrastive Unpaired Translation (CUT), a generative adversarial network (GAN) variant. The CUT model learns the visual characteristics of transparent liquid and applies them to images of colored liquid. Crucially, this translation preserves the underlying geometry—the liquid level and the cup's shape remain identical, while the "style" changes from green to transparent.
3. Automatic Label Generation
Segmenting colored liquid is a trivial task using standard background subtraction. Because the CUT model preserves the spatial structure of the original image, the segmentation mask generated for the colored liquid serves as a perfect "pseudo-ground truth" label for the synthetically generated transparent liquid image.
4. Segmentation Training
With a large dataset of synthetic transparent liquid images and their corresponding labels, the team trained a U-Net architecture. This model learns to identify the subtle visual cues of transparency—such as refraction and surface highlights—to predict a pixel-perfect mask in real-world scenarios.
Experimental Results
The system was evaluated using a Franka Panda robotic arm tasked with pouring water into various containers, utilizing the segmentation to estimate fill levels for closed-loop control.
Segmentation Performance
The model achieved an Intersection over Union (IoU) of 0.72 on transparent liquids. This is a significant achievement for a self-supervised method, as it provides high-fidelity masks without any manual human intervention in the labeling process.
Robotic Pouring Precision
In real-world robotic trials, the system reached target fill levels with an average error of only 0.94% (approximately 0.13 cm in height). Interestingly, the model trained on synthetic transparent data proved more robust than models trained on colored liquid. It learned to ignore transient visual noise, such as splashes and bubbles, which often cause errors in simpler height estimation systems.
Generalization and Robustness
To test the model's versatility, the researchers introduced unseen variables:
- Diverse Containers: The model successfully segmented liquid in jars, square glasses, and containers with spouts.
- Dynamic Backgrounds: By placing an LED screen behind the pouring station, the team simulated various indoor and outdoor environments. The model remained robust, proving it had learned to identify the specific way liquid distorts a background rather than simply memorizing the training environment.
Conclusion
This research demonstrates that self-supervised learning, powered by generative image translation, can effectively solve the problem of transparent liquid perception. By eliminating the need for manual annotations and specialized sensors, this pipeline paves the way for more capable and adaptable service robots. Future work will focus on removing the dependency on initial cup detection and extending the system to estimate volume in more complex, irregularly shaped vessels.
Summary updated