Liquid Content Detection In Transparent Containers: A Benchmark

You Wu, Hengzhou Ye, Yaqing Yang, Zhaodong Wang, Shuiwang Li

TL;DR

This paper introduces a new computer vision task: detecting both the presence of transparent containers and their liquid content level simultaneously. To facilitate research in this area, they release the LCDTC dataset and propose two baseline detectors, LCD-YOLOF and LCD-YOLOX, demonstrating the feasibility of this combined task.

What Problem Does This Paper Address?

Accurately determining the amount of liquid in transparent containers is crucial for many applications, from service robots and industrial monitoring to security checks. Existing computer vision methods typically focus on either detecting the container itself or estimating the liquid level. Container detection alone provides limited information, while precise liquid level estimation is often complex and scenario-specific. This paper addresses the need for a more integrated approach that can provide both container localization and a general estimation of liquid content, making it more practical for real-world scenarios.

Methods

The core of this work is the introduction of a novel task that combines transparent container detection with liquid content estimation. The liquid content is categorized into five states: 'empty', 'little', 'half', 'much', and 'full'. To build their baseline models, the authors adapt two existing identity-preserved human posture detection frameworks, IPH-YOLOF and IPH-YOLOX. They introduce an additional classification head to these models specifically for predicting the liquid content state. To further enhance performance, they incorporate a "triplet attention" mechanism into this head. For the LCD-YOLOX baseline, they also experiment with the CrossFormer backbone, known for its ability to handle cross-scale features. The models are trained using a combined loss function that accounts for classification, bounding box regression, and liquid content prediction.

Datasets

The paper introduces the LCDTC (Liquid Content Detection in Transparent Containers) dataset. This dataset comprises 5916 images, with a 7:3 split between training and testing sets. The images are annotated with axis-aligned bounding boxes for transparent containers and the corresponding liquid content state ('empty', 'little', 'half', 'much', 'full'). The dataset was created by combining internet-sourced images and self-captured images to ensure diversity in scenarios, backgrounds, and container types.

Key Insights and Ideas

  • Task Formulation: The paper argues that combining container detection with a crude liquid content estimation is a more practical and broadly applicable task than precise liquid level estimation. This acknowledges how humans often make such judgments visually.
  • Dataset Necessity: The authors highlight the lack of existing datasets that specifically address this combined task, necessitating the creation of LCDTC.
  • Attention Mechanism Effectiveness: The integration of triplet attention into the liquid content prediction head demonstrably improves the performance of both baseline detectors.
  • Backbone Impact: For LCD-YOLOX, using the CrossFormer backbone, particularly the -S variant, shows significant improvements over the original CSPDarknet backbone, suggesting the benefit of advanced feature extraction for this task.
  • Category vs. Content Difficulty: The results indicate that detecting the transparent container itself is generally easier than accurately estimating its liquid content, likely due to the inherent visual challenges of liquids (reflections, lack of fixed shape, and subtle visual cues).

Key Results

  • Baseline Performance: The proposed LCD-YOLOX detector consistently outperformed LCD-YOLOF across various metrics. For instance, LCD-YOLOX achieved higher mAP values for liquid content detection (mAPł) compared to LCD-YOLOF.
  • Triplet Attention Gains: Incorporating triplet attention led to noticeable improvements in mAPc, mAPł, and mAPct for both detectors. For LCD-YOLOX, these gains were around 0.3-3.0%.
  • CrossFormer Backbone Gains: Using the CrossFormer-S backbone with LCD-YOLOX resulted in improvements, with mAPc reaching 0.704 and mAPł reaching 0.533.
  • Liquid Content State Performance: Both detectors performed best on the 'little' and 'empty' liquid content states. 'Half', 'much', and 'full' states were more challenging, likely due to greater intra-class variations and visual ambiguities.
  • Qualitative Results: While the models performed well on clear images, they struggled with color interference, non-standard container sizes, and occlusions, leading to misclassifications or missed detections.

Limitations and Future Work

  • Performance Gap: The authors acknowledge that the current performance of their baseline models is "still not satisfying," particularly for the more challenging liquid content states.
  • Counteracting Effects: They identify a potential "counteracting effect" between the tasks of container localization and liquid content estimation when performed simultaneously. Future work will focus on developing methods to mitigate this.
  • Dataset Imbalance: While the dataset aims for balanced distribution, the inherent real-world frequency of certain liquid states leads to some imbalance, which the authors believe their experimental results have effectively addressed. However, they note that the dataset currently excludes very small objects, where determining liquid content is particularly difficult.
  • Future Directions: The authors plan to explore improved techniques to enhance performance and address the counteracting effects between the two sub-tasks. They also aim to inspire further research into this challenging area.

Summary updated

Abstract

Various substances that possess liquid states include drinking water, various types of fuel, pharmaceuticals, and chemicals, which are indispensable in our daily lives. There are numerous real-world applications for liquid content detection in transparent containers, for example, service robots, pouring robots, security checks, industrial observation systems, etc. However, the majority of the existing methods either concentrate on transparent container detection or liquid height estimation; the former provides very limited information for more advanced computer vision tasks, whereas the latter is too demanding to generalize to open-world applications. In this paper, we propose a dataset for detecting liquid content in transparent containers (LCDTC), which presents an innovative task involving transparent container detection and liquid content estimation. The primary objective of this task is to obtain more information beyond the location of the container by additionally providing certain liquid content information which is easy to achieve with computer vision methods in various open-world applications. This task has potential applications in service robots, waste classification, security checks, and so on. The presented LCDTC dataset comprises 5916 images that have been extensively annotated through axis-aligned bounding boxes. We develop two baseline detectors, termed LCD-YOLOF and LCD-YOLOX, for the proposed dataset, based on two identity-preserved human posture detectors, i.e., IPH-YOLOF and IPH-YOLOX. By releasing LCDTC, we intend to stimulate more future works into the detection of liquid content in transparent containers and bring more focus to this challenging task.
BibTeX
@inproceedings{wu2023liquid,
  author = {You Wu and Hengzhou Ye and Yaqing Yang and Zhaodong Wang and Shuiwang Li},
  title = {{Liquid Content Detection In Transparent Containers: A Benchmark}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=p8q2bpHBvM},
}

Loading related content...

Figure