Advancing from Automated to Autonomous Beamline by Leveraging Computer Vision

Baolu Li, Hongkai Yu, Huiming Sun, Jin Ma, Yuewei Lin, Lu Ma, Yonghua DuarXiv 2025 · DOI · arXiv

Introduction

Synchrotron light sources are massive, multi-million dollar "super-microscopes" used to study materials at the atomic level. While many beamlines are currently "automated"—meaning robots swap samples and collect data—they lack true autonomy. The primary bottleneck is safety: current systems still require a human operator to monitor video feeds to prevent robotic arms from colliding with fragile detectors. Existing hardware safety measures, such as limit switches or predefined motion paths, are static and cannot account for dynamic variables like fallen samples or custom experimental setups.

Conceptual shift from human-monitored automated beamlines to AI-driven autonomous beamlines.

System Architecture

To achieve 24/7 autonomous operation, researchers have developed a "Computer Vision (CV) Stack" designed to replace human oversight with real-time collision awareness. The system follows a four-stage pipeline:

  1. Multi-view Data Collection: Three high-resolution cameras are positioned 120 degrees apart to eliminate blind spots and occlusions.
  2. Segmentation & Tracking: The system utilizes YOLOv8 for instance segmentation to identify equipment (e.g., "Holder," "Stage") and Associating Objects with Transformers (AOT) to maintain object identity across frames.
  3. Geometry Regularization: Domain knowledge is injected into the model as a "sanity check." Since the beamline is a controlled environment, spatial truths (e.g., the holder's position relative to the stage) act as mathematical constraints to improve robustness.
  4. Collision Monitoring: The system calculates the minimum Euclidean distance between equipment contours. If the distance drops below a 10-pixel threshold across all views, an immediate stop command is triggered.
The proposed system architecture, illustrating the flow from the beamline environment through the computer vision pipeline to real-time feedback.

Training and Implementation

The system was developed using a transfer learning approach. Models were initially trained on a Simulated Beamline Dataset (870 frames) in a controlled lab setting and subsequently fine-tuned on the Real QAS BNL Beamline Dataset. This new benchmark consists of 1,100 annotated frames from the Quick X-ray Absorption Spectroscopy (QAS) beamline at Brookhaven National Laboratory, capturing diverse scenarios including "near-miss" collisions.

A key innovation is the "Human-in-the-loop" annotation module. Using the Segment Anything Model (SAM), scientists can quickly annotate new or custom tools by clicking on them in the software. The system then automatically generates training masks, allowing the AI to adapt to new experimental hardware on the fly.

Performance Results

The multi-view approach proved essential; while single-camera systems are prone to false positives due to occlusions, the integrated system achieved 99.8% accuracy in collision detection. Segmentation quality was high, with a Mean Intersection over Union (mIoU) of 90.9% in the top-view.

In terms of real-time performance, the system operates at 6 Frames Per Second (FPS) on a consumer-grade GPU (NVIDIA RTX 3090). Given the relatively slow movement of beamline equipment, the authors estimate that 10 FPS—achievable with multi-GPU setups—is sufficient for reliable safety. Theoretically, at this performance level, the system would experience a detection error only once every 198 years.

Conclusion

While highly effective, the system faces challenges regarding computational load and the extreme safety requirements of national facilities. Future work will focus on optimizing the pipeline for higher frame rates and implementing hardware redundancy—running parallel AI systems to reduce error probability to near-zero. Expanding testing to different beamline types (e.g., imaging vs. scattering) will further validate the system's generalizability across diverse scientific environments.

Summary updated

Abstract

The synchrotron light source, a cutting-edge large-scale user facility, requires autonomous synchrotron beamline operations, a crucial technique that should enable experiments to be conducted automatically, reliably, and safely with minimum human intervention. However, current state-of-the-art synchrotron beamlines still heavily rely on human safety oversight. To bridge the gap between automated and autonomous operation, a computer vision-based system is proposed, integrating deep learning and multiview cameras for real-time collision detection. The system utilizes equipment segmentation, tracking, and geometric analysis to assess potential collisions with transfer learning that enhances robustness. In addition, an interactive annotation module has been developed to improve the adaptability to new object classes. Experiments on a real beamline dataset demonstrate high accuracy, real-time performance, and strong potential for autonomous synchrotron beamline operations.
BibTeX
@misc{li2025advancing,
  author = {Baolu Li and Hongkai Yu and Huiming Sun and Jin Ma and Yuewei Lin and Lu Ma and Yonghua Du},
  title = {{Advancing from Automated to Autonomous Beamline by Leveraging Computer Vision}},
  year = {2025},
  eprint = {2506.00836},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.00836},
}
Figure