Abstract
Autonomous scientific discovery requires robotic laboratories capable of self-correction. A primary hurdle is visual anomaly detection, which is often context-dependent—a state may be normal or anomalous depending on the specific experimental subtask. This paper presents a novel framework leveraging Vision-Language Models (VLMs) and a "hierarchical prompting" strategy to provide procedural context. By structuring prompts into four levels of increasing detail, the method achieves up to 79.2% accuracy in identifying experimental faults using GPT-4o, significantly outperforming traditional vision methods and less capable models.
Introduction
In the pursuit of fully automated "self-driving" laboratories, robots are tasked with complex procedures such as handling hazardous chemicals and manipulating delicate glassware. While automation increases throughput, it also introduces various failure modes: missing test tubes, stuck lids, or failed liquid transfers.
Traditional computer vision approaches often treat anomaly detection as a static image-matching problem. However, in a laboratory setting, anomalies are inherently context-dependent. For instance, a robot arm with an empty gripper is expected before a pick-up task but represents a failure after the task should have been completed. This research addresses the need for a reasoning-based approach that integrates visual data with procedural knowledge to understand the current state of a scientific workflow.
Methodology: Hierarchical Prompting
The core of the proposed framework is a hierarchical prompting strategy designed to bridge the gap between raw visual input and complex experimental logic. Rather than providing a single flat description, the system provides information across four distinct levels:
- Level 1 (Experiment Context): General background of the laboratory environment.
- Level 2 (Stage Description): Details regarding the specific subtask currently in progress (e.g., "transferring a sample").
- Level 3 (Detection Content): Explicit instructions on what the model should verify (e.g., "check if the test tube is present in the rack").
- Level 4 (Anomaly Label Description): Formal definitions of "normal" and "abnormal" states for the specific step.
By utilizing Chain-of-Thought (CoT) reasoning, the VLM is prompted to explain its logic step-by-step before arriving at a final classification. This ensures that the model's conclusion is grounded in the provided procedural context.
Experimental Setup and Dataset
To evaluate the framework, the researchers developed a specialized benchmark based on a Silicone Preparation Workflow in a real chemical laboratory. The dataset consists of 1,001 first-person images (501 normal, 500 abnormal) captured from 20 different monitoring points. These points include both fixed-position cameras and cameras mounted on mobile robotic arms, providing a diverse range of perspectives. Each image is paired with structured textual descriptions corresponding to the four hierarchical prompt levels.
Key Results and Discussion
The study compared the performance of GPT-4o and Qwen2.5-VL-72B-Instruct. The results highlighted several critical insights:
- Scaling with Context: GPT-4o's accuracy improved dramatically as the prompt detail increased, rising from 41.6% at Level 1 to 79.2% at Level 4.
- The Importance of Detection Objectives: A significant performance spike occurred at Level 3, suggesting that explicitly telling the model what to look for is more critical than general stage descriptions.
- Safety and Reliability: At the highest prompt level, GPT-4o achieved a Missed Detection Rate (MDR) of only 3.0%, indicating a high degree of reliability in catching actual experimental errors.
- Model Reasoning Depth: GPT-4o significantly outperformed Qwen2.5-VL, which suffered from a high False Positive Rate (35.6% at Level 4). This suggests that the underlying reasoning depth of the VLM is a primary bottleneck for complex laboratory tasks.
Conclusion
This research demonstrates that VLMs, when guided by hierarchical context, can serve as powerful reasoning engines for laboratory monitoring. While the current False Positive Rate (16.8% for GPT-4o) remains a challenge for high-throughput environments, the system's zero-shot capability allows it to be deployed across diverse experiments without the need for task-specific fine-tuning. Future work will focus on automating prompt generation from Standard Operating Procedures (SOPs) and integrating multi-modal sensor data to further refine detection accuracy.
Summary updated