Affordable Artificial Intelligence-Assisted Machine Supervision System for the Small and Medium-Sized Manufacturers

Chen Li, Shijie Bian, Tongzi Wu, Richard P. Donovan, Bingbing Li2022 · DOI

TL;DR

This paper introduces the AI-assisted Machine Supervision (AIMS) system, an affordable and scalable solution designed to bring Industry 4.0 capabilities to Small and Medium-sized Manufacturers (SMMs). AIMS leverages consumer-grade electronics and AI to provide real-time machine monitoring and human-machine interaction analysis, aiming to boost productivity and reduce costs for SMMs.

What Problem Does This Paper Address?

The rapid advancement of Industry 4.0, driven by AI and IoT, promises significant improvements in manufacturing efficiency, quality, and safety. However, the high capital costs and complexity associated with implementing these advanced IT infrastructures pose a major barrier for Small and Medium-sized Manufacturers (SMMs). These businesses often rely heavily on manual supervision, lacking the resources to gain real-time visibility into factory operations, equipment status, and workflow. This paper addresses the need for an affordable, accessible, and portable solution that automates machine supervision for SMMs, bridging the gap between advanced manufacturing technologies and the practical constraints of smaller enterprises.

Methods

The AIMS system is comprised of two main subsystems:

  • Direct Machine Monitoring (DMM): This subsystem uses computer vision to monitor the motion of major machine components in real-time. It doesn't rely on direct machine data but instead analyzes visual cues to infer the machine's operational state. Key to this is the creation of a machine state transition diagram, which graphically represents operational states and constraints. AIMS uses adaptive AI training to handle variations between individual machines. When a component's motion deviates from the expected state or violates constraints, the system detects an anomaly.
  • Human-Machine Interaction Monitoring (HIM): This subsystem observes human interaction with the machine from an external perspective. It uses a wearable camera to capture the worker's view, focusing on the machine's control panel. Advanced computer vision techniques, including text detection and recognition (using the CRAFT model) and finger detection, are employed to understand the worker's inputs and the machine's feedback. This allows AIMS to correlate human actions with machine responses, ensuring that interactions have the expected impact and contributing to safety and cybersecurity.

The system is designed to be deployed on affordable embedded devices, leveraging cloud computing for model training.

Datasets

The paper describes the creation of a custom dataset for training and evaluation.

  • 3D Printer Dataset: Approximately 1500 images were collected from a 3D printer operating under various lighting and perspective conditions. These images were annotated with bounding boxes for key components (extruder, build plate, motor axis) for DMM training. For HIM, images captured during human-machine interaction were used, focusing on display screens and buttons.

Key Insights and Ideas

  • Affordability through Consumer-Grade Tech: The core innovation lies in using readily available, low-cost hardware (webcams, embedded VPUs like NVIDIA Jetson) and avoiding expensive proprietary protocols. This makes advanced monitoring accessible to SMMs.
  • Vision-Based Supervision without Direct Machine Integration: AIMS demonstrates that valuable machine state information can be inferred purely from visual observation of component movement, bypassing the need for complex and costly integration with machine control systems.
  • State Transition Diagrams for Efficiency: The use of state transition diagrams significantly reduces computational complexity and improves the accuracy of state change detection compared to more general error-checking methods.
  • Human-Machine Interaction as a Critical Component: Recognizing that human operators are integral to many SMM processes, the HIM subsystem adds a crucial layer of monitoring, ensuring that human actions are correctly interpreted and contribute to overall system understanding.
  • Scalability and Portability: The system's design emphasizes ease of deployment and adaptability, allowing it to be scaled to different machines and environments with minimal modifications.

Key Results

  • DMM Performance: The DMM model achieved high average precision (AP) on the custom 3D printer dataset, with results around 99.8% for the extruder, build plate, and axis components at an IoU threshold of 0.5. The system achieved processing speeds of 2.3 FPS on NVIDIA Jetson Nano and 16.2 FPS on Jetson Xavier NX.
  • HIM Performance: The text detection component of the HIM module achieved accuracies up to 97.35% under optimal lighting conditions. The finger detection component achieved an average accuracy of approximately 89.87% in identifying button regions, with 100% accuracy in detecting which button was interacted with once the region was correctly identified.
  • Overall AIMS Performance: In normal operation tests, AIMS demonstrated outstanding performance, with only a few frames (out of ten tasks) showing incorrect predictions, which were effectively handled by its 5-frame buffer mechanism. In abnormal condition tests, AIMS successfully detected anomalies within five frames in 9 out of 10 jobs. The overall numerical results for 19 tests showed a precision of 0.923 and a recall of 0.936.

Limitations and Future Work

  • Limitations:
    • Robustness to Extreme Cases: The algorithm is not fully robust to extreme cases where consecutive frames follow a distribution not seen in the training data, which can be caused by unstable lighting, unexpected blockages, or vibrations.
  • Training Data Requirements: The DMM component requires a significant amount of manually labeled training data, which can be time-consuming and labor-intensive to acquire.
  • Halt Test Anomaly: The system did not flag a "halt test" (machine stopped at a calibration state) as an anomaly because the components remained within their legitimate range of motion.
  • Future Work:
    • Expanded Use Cases: Exploring more use cases beyond 3D printing and assembly lines, with different interfaces and human-machine interactions, to enhance the system's generality.
  • Interactive AR Training: Developing interactive Augmented Reality (AR) training modules that leverage finger detection and text recognition algorithms through AR goggles.

Summary updated

BibTeX
@misc{li2022affordable,
  author = {Chen Li and Shijie Bian and Tongzi Wu and Richard P. Donovan and Bingbing Li},
  title = {{Affordable Artificial Intelligence-Assisted Machine Supervision System for the Small and Medium-Sized Manufacturers}},
  year = {2022},
  doi = {10.3390/s22166246},
}

Loading related content...

Figure