Autonomous Incident Detection on Spectrometers Using Deep Convolutional Models

Xuelin Zhang, Donghao Zhang, Alexander Leye, Adrian Scott, Luke Visser, Zongyuan Ge, C. Paul Bonnington

TL;DR

This paper introduces a novel deep learning system for automatically detecting two critical incidents – liquid beading and flooding – in spectrometers used in analytical chemistry. By combining advanced convolutional neural networks with domain expertise and custom data augmentation, the system achieves high accuracy (95% for beads, 98% for flooding) in real-time, preventing instrument damage and ensuring reliable analysis.

What Problem Does This Paper Address?

Scientific instruments like spectrometers, crucial for analytical chemistry, rely on precise sample introduction. However, issues like liquid beading (formation of small, transparent droplets) and flooding (excess liquid accumulation) on the spray chamber can lead to inaccurate results, instrument malfunction, and even damage. These incidents are challenging to detect automatically because the involved objects are small, transparent, and deformable, making them difficult for standard computer vision models. Current solutions often require constant human monitoring, which is time-consuming and labor-intensive. This paper aims to bridge this gap by developing an autonomous, computer vision-based system for real-time incident detection in spectrometers.

Methods

The researchers developed two distinct frameworks, one for flooding detection and another for bead detection, both leveraging deep convolutional models.

For flooding detection, a two-stage approach was employed. The first stage uses a Mask R-CNN model to identify bounding boxes for the flooding and the drain tube. The second stage incorporates a temporal model that analyzes time-series data. This temporal model uses the confidence scores from the Mask R-CNN and the mean squared error (MSE) of pixel differences in the drain tube region between consecutive frames. This temporal aspect is crucial for handling the deformable nature of flooding and improving detection consistency, as flooding is a time-series event.

For bead detection, the team focused on optimizing object detection components. They selected HRNetV2p as the backbone and Hybrid Task Cascade (HTC) as the detector. To address the challenges of small, transparent, and deformable beads, they implemented several techniques:

  • Data Augmentation and Synthesis: Techniques like Cut-Paste augmentation were used to create synthetic images by pasting augmented beads onto clean scenes, increasing the dataset's robustness and variety.
  • Hard Negative Mining (OHEM): This sampler helps the model learn to distinguish between actual beads and similar-looking background elements by focusing on difficult-to-classify samples.
  • Expert Knowledge Integration: The choice of backbone and detector architecture was guided by the specific challenges of detecting small and transparent objects.

Datasets

Since no existing datasets were suitable for this specific task, the researchers created their own dataset from a chemistry lab environment.

  • Bead Detection Dataset: Collected using GoPro7 and Basler acA2440 cameras, capturing images and videos from various angles and under different lighting conditions. This dataset includes annotations for spray chamber, nebulizer, drain tube, and beading. Synthetic data was generated to augment the original dataset.
  • Flooding Detection Dataset: Collected using a Basler acA2440 camera, providing black and white images from consecutive runs. This dataset includes annotations for spray chamber, drain tube, flooding, and bubbles.

Key Insights and Ideas

  • Temporal Information is Key for Flooding: Detecting flooding solely from static images is difficult due to its deformable nature. Incorporating temporal information, specifically the movement of bubbles in the drain tube and the history of flooding confidence scores, significantly improves detection accuracy and consistency.
  • Transparency and Small Size are Major Challenges: The inherent properties of beads (transparency, small size) make them exceptionally difficult for standard object detectors, which often rely on texture and distinct edges.
  • Domain Expertise Enhances Deep Learning: Simply applying off-the-shelf deep learning models wasn't sufficient. Integrating expert knowledge about the physical phenomena (e.g., bubble movement for flooding, the need for robust feature representation for small beads) was crucial for designing effective models.
  • Data Augmentation and Negative Mining are Essential for Small Objects: For bead detection, techniques like Cut-Paste synthesis and hard negative mining were vital for improving robustness and reducing false positives, especially given the limited initial dataset.

Key Results

  • Flooding Detection: The proposed temporal model achieved a precision of 0.985 and a recall of 0.993, significantly outperforming a standalone Mask R-CNN model (AP of 0.603).
  • Bead Detection: The optimized framework, using HRNetV2p as the backbone and HTC as the detector with OHEM and augmentation, achieved a mean Average Precision (mAP) of 83.9%. This performance surpassed other state-of-the-art object detection baselines.
  • Real-time Performance: The system can process four frames per second, making it suitable for real-time implementation in laboratory settings.

Limitations and Future Work

  • Lighting Sensitivity: The detection of transparent objects is affected by lighting conditions. The authors suggest collecting data under diverse lighting and exploring data augmentation techniques like contrast adjustment to mitigate this.
  • Camera Angle Dependency: The current model is trained on data from a fixed camera position. Future work includes testing and training on data from different camera angles to improve robustness and generalizability.
  • 3D Information for Enhanced Robustness: Incorporating 3D CAD models and depth information could further enhance bead detection robustness.
  • Broader Applicability: The core techniques for detecting liquid-based incidents could be extended to other laboratory apparatus and potentially other types of incidents like leakage or pressure anomalies.
  • External Dataset Validation: Assessing the model's performance on an external dataset would validate its generalizability beyond the specific spectrometer context.

Summary updated

Abstract

This paper focuses on improving the performance of scientific instrumentation that uses glass spray chambers for sample introduction, such as spectrometers, which are widely used in analytical chemistry, by detecting incidents using deep convolutional models. The performance of these instruments can be affected by the quality of the introduction of the sample into the spray chamber. Among the indicators of poor quality sample introduction are two primary incidents: The formation of liquid beads on the surface of the spray chamber, and flooding at the bottom of the spray chamber. Detecting such events autonomously as they occur can assist with improving the overall operational accuracy and efficacy of the chemical analysis, and avoid severe incidents such as malfunction and instrument damage. In contrast to objects commonly seen in the real world, beading and flooding detection are more challenging since they are of significantly small size and transparent. Furthermore, the non-rigid property increases the difficulty of the detection of these incidents, as such that existing deep-learning-based object detection frameworks are prone to fail for this task. There is no former work that uses computer vision to detect these incidents in the chemistry industry. In this work, we propose two frameworks for the detection task of these two incidents, which not only leverage the modern deep learning architectures but also integrate with expert knowledge of the problems. Specifically, the proposed networks first localize the regions of interest where the incidents are most likely generated and then refine these incident outputs. The use of data augmentation and synthesis, and choice of negative sampling in training, allows for a large increase in accuracy while remaining a real-time system for inference. In the data collected from our laboratory, our method surpasses widely used object detection baselines and can correctly detect 95% of the beads and 98% of the flooding. At the same time, out method can process four frames per second and is able to be implemented in real time.
BibTeX
@inproceedings{zhang2022autonomous,
  author = {Xuelin Zhang and Donghao Zhang and Alexander Leye and Adrian Scott and Luke Visser and Zongyuan Ge and C. Paul Bonnington},
  title = {{Autonomous Incident Detection on Spectrometers Using Deep Convolutional Models}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=165udN7PHlP},
}
Figure