SLURP! Spectroscopy of Liquids Using Robot Pre-Touch Sensing

Nathaniel Hanson, Wesley Lewis, Kavya Puthuveetil, Donelle Furline, Akhil Padmanabha, Taskin Padir, Zackory Erickson

TL;DR

Robots struggle to identify liquids in opaque containers. This paper introduces SLURP, a novel system that uses Visible to Near Infrared (VNIR) spectroscopy integrated into a robot's gripper to non-destructively identify both the container material and its liquid contents with high accuracy. This breakthrough enables robots to better handle and sort household items without needing to open or touch them.

What Problem Does This Paper Address?

In domestic environments, robots will encounter countless containers holding various liquids and granular materials. Identifying these contents is crucial for safe and effective manipulation, whether it's sorting recycling, preparing food, or managing medicine. Traditional robot vision (cameras, LiDAR) is limited by container opacity, and tactile sensing struggles with the wide range of liquid densities and viscosities. Opening containers to directly observe contents is a complex dexterous manipulation task that still requires subsequent sensing. This paper tackles the fundamental challenge: can a robot reliably identify what's inside a container without opening or touching it?

Methods

The core of the SLURP system is the integration of Visible to Near Infrared (VNIR) spectroscopy directly into a robot's end-effector. Specifically, the researchers developed a custom gripper with two spectrometers: a Hamamatsu C12880ma for the visible range (340-850 nm) and a Mantispectra OEM Board for the near-infrared range (850-1700 nm). This dual-spectrometer approach covers a broad spectral range, crucial for distinguishing materials.

The system works by illuminating the container with a Quartz Tungsten Halogen (QTH) bulb. Light interacts with the container and its contents, and the reflected/scattered light is captured by the spectrometers. A key innovation is the use of a white Teflon sheet on the opposing gripper paddle, acting as a spectralon-like reference to maximize signal return for highly transmissive liquids.

To process the spectral data, they employ a hierarchical modeling approach. This involves two Multi-Layer Perceptrons (MLPs): one to predict the container material and another to predict the internal contents. Crucially, the container's predicted class is fed as an input to the content prediction MLP, allowing the model to leverage container information for more accurate content identification. They also incorporate the gradient of the spectral signal as an input feature, which proved beneficial.

Datasets

The authors created and open-sourced a novel dataset specifically for this research. This dataset comprises spectral measurements from over 180 unique combinations of containers and liquids. They collected 30 spectral samples for each combination, resulting in a total of 5,070 spectral measurements per spectrometer. The containers varied in material (glass, polypropylene, PET, paper, acrylic, silicone), color, clarity, geometry, and thickness. The contents included 13 classes of liquids and granular solids, chosen for their commonality and difficulty in distinguishing via RGB vision (e.g., sugar vs. salt, various medicines).

Key Insights and Ideas

  • VNIR Spectroscopy as a Non-Contact Identifier: The most significant insight is that VNIR spectroscopy, when integrated into a robot's gripper, can effectively "see" through opaque containers to identify their contents without physical interaction. This opens up a new paradigm for robot perception in domestic settings.
  • Hierarchical Modeling for Context: The hierarchical approach, where container prediction informs content prediction, significantly improves accuracy. This highlights the importance of contextual information in spectral analysis for complex scenarios.
  • Container Material Matters: The research found that container material significantly influences the spectral signature. The model achieved higher accuracy in identifying containers (96%) than contents (85%), underscoring the container's role as a crucial intermediary in the spectral signal.
  • Broad Spectral Range is Key: Combining visible and near-infrared spectrometers was essential. The paper notes that using either spectrometer alone resulted in substantially lower accuracy, emphasizing the benefit of a wider spectral coverage for material discrimination.

Key Results

The SLURP system demonstrated impressive performance:

  • Content Identification: Over 85% accuracy in identifying 13 different liquids and granular media within 13 different containers.
  • Container Identification: 96% accuracy in identifying the material composition of the containers themselves.
  • Hierarchical vs. Naïve Models: The hierarchical model outperformed a naïve implementation (two disjoint networks) by 8% for container prediction and 13% for content prediction.
  • Robot Experiment Performance: In real-time experiments with a Stretch RE1 robot, the system achieved overall container and substrate classification accuracies of 78% and 72%, respectively, for a subset of challenging container-substrate combinations.

Limitations and Future Work

  • Distinguishing Similar Substances: The paper acknowledges that distinguishing between very similar substances (e.g., sugar vs. salt, coke vs. soy sauce) remains challenging, likely due to their similar optical properties and lower signal returns.
  • Environmental Illumination: While efforts were made to mitigate ambient light, the system's performance could still be affected by significant variations in environmental illumination.
  • Future Work: The authors suggest exploring more advanced deep learning architectures, incorporating additional sensor modalities, and expanding the dataset to include a wider variety of materials and conditions. They also aim to improve the robustness of the system to environmental changes and further refine the ability to distinguish between highly similar substances.

Summary updated

Abstract

Liquids and granular media are pervasive throughout human environments. Their free-flowing nature causes people to constrain them into containers. We do so with thousands of different types of containers made out of different materials with varying sizes, shapes, and colors. In this work, we present a state-of-the-art sensing technique for robots to perceive what liquid is inside of an unknown container. We do so by integrating Visible to Near Infrared (VNIR) reflectance spectroscopy into a robot's end effector. We introduce a hierarchical model for inferring the material classes of both containers and internal contents given spectral measurements from two integrated spectrometers. To train these inference models, we capture and open source a dataset of spectral measurements from over 180 different combinations of containers and liquids. Our technique demonstrates over 85% accuracy in identifying 13 different liquids and granular media contained within 13 different containers. The sensitivity of our spectral readings allow our model to also identify the material composition of the containers themselves with 96% accuracy. Overall, VNIR spectroscopy presents a promising method to give household robots a general-purpose ability to infer the liquids inside of containers, without needing to open or manipulate the containers.
BibTeX
@inproceedings{hanson2023slurp,
  author = {Nathaniel Hanson and Wesley Lewis and Kavya Puthuveetil and Donelle Furline and Akhil Padmanabha and Taskin Padir and Zackory Erickson},
  title = {{SLURP! Spectroscopy of Liquids Using Robot Pre-Touch Sensing}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10161084},
}
Figure