A VLM-based Method for Visual Anomaly Detection in Robotic Scientific Laboratories

Huaping Liu(刘华平), Shiwei Lin, Chenxu Wang, Xiaozhen Ding, Yi Wang, Boyuan Du, Lei Song, Chenggang Wang, Huaping LiuarXiv 2025 · DOI · arXiv

Abstract

Autonomous scientific discovery requires robotic laboratories capable of self-correction. A primary hurdle is visual anomaly detection, which is often context-dependent—a state may be normal or anomalous depending on the specific experimental subtask. This paper presents a novel framework leveraging Vision-Language Models (VLMs) and a "hierarchical prompting" strategy to provide procedural context. By structuring prompts into four levels of increasing detail, the method achieves up to 79.2% accuracy in identifying experimental faults using GPT-4o, significantly outperforming traditional vision methods and less capable models.

Introduction

In the pursuit of fully automated "self-driving" laboratories, robots are tasked with complex procedures such as handling hazardous chemicals and manipulating delicate glassware. While automation increases throughput, it also introduces various failure modes: missing test tubes, stuck lids, or failed liquid transfers.

Traditional computer vision approaches often treat anomaly detection as a static image-matching problem. However, in a laboratory setting, anomalies are inherently context-dependent. For instance, a robot arm with an empty gripper is expected before a pick-up task but represents a failure after the task should have been completed. This research addresses the need for a reasoning-based approach that integrates visual data with procedural knowledge to understand the current state of a scientific workflow.

Context-dependent anomaly detection: The same visual state can be interpreted as normal, anomalous, or uncertain depending on the experimental stage and subtask context.

Methodology: Hierarchical Prompting

The core of the proposed framework is a hierarchical prompting strategy designed to bridge the gap between raw visual input and complex experimental logic. Rather than providing a single flat description, the system provides information across four distinct levels:

  1. Level 1 (Experiment Context): General background of the laboratory environment.
  2. Level 2 (Stage Description): Details regarding the specific subtask currently in progress (e.g., "transferring a sample").
  3. Level 3 (Detection Content): Explicit instructions on what the model should verify (e.g., "check if the test tube is present in the rack").
  4. Level 4 (Anomaly Label Description): Formal definitions of "normal" and "abnormal" states for the specific step.
The VLM-based visual reasoning framework, illustrating the flow from visual acquisition to hierarchical reasoning and final decision-making.

By utilizing Chain-of-Thought (CoT) reasoning, the VLM is prompted to explain its logic step-by-step before arriving at a final classification. This ensures that the model's conclusion is grounded in the provided procedural context.

Experimental Setup and Dataset

To evaluate the framework, the researchers developed a specialized benchmark based on a Silicone Preparation Workflow in a real chemical laboratory. The dataset consists of 1,001 first-person images (501 normal, 500 abnormal) captured from 20 different monitoring points. These points include both fixed-position cameras and cameras mounted on mobile robotic arms, providing a diverse range of perspectives. Each image is paired with structured textual descriptions corresponding to the four hierarchical prompt levels.

Key Results and Discussion

The study compared the performance of GPT-4o and Qwen2.5-VL-72B-Instruct. The results highlighted several critical insights:

  • Scaling with Context: GPT-4o's accuracy improved dramatically as the prompt detail increased, rising from 41.6% at Level 1 to 79.2% at Level 4.
  • The Importance of Detection Objectives: A significant performance spike occurred at Level 3, suggesting that explicitly telling the model what to look for is more critical than general stage descriptions.
  • Safety and Reliability: At the highest prompt level, GPT-4o achieved a Missed Detection Rate (MDR) of only 3.0%, indicating a high degree of reliability in catching actual experimental errors.
  • Model Reasoning Depth: GPT-4o significantly outperformed Qwen2.5-VL, which suffered from a high False Positive Rate (35.6% at Level 4). This suggests that the underlying reasoning depth of the VLM is a primary bottleneck for complex laboratory tasks.

Conclusion

This research demonstrates that VLMs, when guided by hierarchical context, can serve as powerful reasoning engines for laboratory monitoring. While the current False Positive Rate (16.8% for GPT-4o) remains a challenge for high-throughput environments, the system's zero-shot capability allows it to be deployed across diverse experiments without the need for task-specific fine-tuning. Future work will focus on automating prompt generation from Standard Operating Procedures (SOPs) and integrating multi-modal sensor data to further refine detection accuracy.

Summary updated

BibTeX
@misc{liu2025vlm,
  author = {Huaping Liu(刘华平) and Shiwei Lin and Chenxu Wang and Xiaozhen Ding and Yi Wang and Boyuan Du and Lei Song and Chenggang Wang and Huaping Liu},
  title = {{A VLM-based Method for Visual Anomaly Detection in Robotic Scientific Laboratories}},
  year = {2025},
  eprint = {2506.05405},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.05405},
}
Figure