Abstract
To improve the efficiency of human-machine interaction in complex machining environments and optimize the accuracy of gesture recognition, a new gesture recognition system is developed by combining the improved You Only Look Once 5 and Unity 3D software. Firstly, an efficient channel attention mechanism is introduced to optimize the network structure of the fifth version of the algorithm to process higher dimensional gesture image data. Secondly, a twin model of complex processing equipment is constructed, and real-time visualization of gesture data and human-machine interaction are achieved using Unity 3D. The research results indicated that the designed static gesture recognition algorithm achieved image signal-to-noise ratio and image intersection to union ratio of 0.95 and 0.98 during the training process. In practical applications, the gesture interaction recognition model designed using this algorithm exhibited extremely low response time, with a minimum of 0.02s to complete the recognition task. At the same time, the recognition accuracy of this model reached up to 99.1%, which was much higher than the other three comparative models. In the practical performance tests, for the different four datasets, the recognition accuracy of YOLOv5-ECA model was 98.5%, 98.7%, 99.1% and 98.8%, with the recognition time as low as 0.07s, 0.02s, 0.11s and 0.08s, respectively. It can be seen that the gesture recognition system provides a new technical solution for human-machine interaction of complex processing equipment, which can further improve the operational efficiency and safety of human-machine interaction.
Introduction
In modern "Smart Factories," human-machine interaction (HMI) often acts as a bottleneck. Traditional control methods, such as physical buttons or touchscreens, can be inefficient, prone to wear, and potentially unsafe in complex machining environments near CNC machines or robotic arms. While gesture recognition offers a touchless, intuitive alternative, existing deep learning models frequently struggle with the "high-dimensional" nature of gesture data in noisy industrial settings. They often lack either the real-time speed required for safety or the precision needed to distinguish subtle hand movements. Furthermore, a "visualization gap" often exists between the user’s gesture and the machine’s state.
Methods and Materials
The proposed solution integrates an advanced detection algorithm with a virtual simulation environment.
Data Acquisition and Preprocessing
The system utilizes Kinect 2.0 sensors to capture both color (RGB) and depth information. This dual-stream data facilitates better segmentation of the hand from complex backgrounds.
The study primarily utilized the EgoHands dataset, containing 3,000 images for first-person perspective gesture recognition across various lighting conditions and diverse users. Images were normalized to 416×416 pixels and augmented with random rotations and horizontal flipping to enhance model robustness.
YOLOv5-ECA Architecture
The core of the system is the YOLOv5-ECA algorithm. The researchers integrated an Efficient Channel Attention (ECA) module into the standard YOLOv5 architecture. Unlike traditional Squeeze-and-Excitation (SE) blocks that reduce dimensionality and lose information, ECA employs one-dimensional convolutions to capture local cross-channel interactions. This allows the model to "focus" on critical gesture features without significantly increasing computational costs.
Unity 3D Digital Twin
To bridge the gap between recognition and action, a digital twin of the machining equipment was built in Unity 3D. Gestures are mapped to virtual commands, providing the operator with real-time visual feedback before the physical machine executes the command.
Results and Discussion
The YOLOv5-ECA model significantly outperformed baseline models like SSD, standard YOLOv5, and EMAFF-Net.
- Accuracy: The model reached a peak accuracy of 99.1%, compared to 90.3% for standard YOLOv5 and 88.2% for SSD.
- Speed: It achieved a minimum response time of 0.02s, meeting the requirements for "instantaneous" human perception.
- Robustness: Across four different gesture sets, accuracy remained consistently high, ranging from 98.5% to 99.1%.
- Training Quality: The model achieved an Image Signal-to-Noise Ratio (SNR) of 0.95 and an Intersection over Union (IoU) of 0.98.
A key insight from the study is that dimensionality preservation in attention mechanisms is crucial. By using 1D convolutions in the ECA module, the authors preserved subtle gesture features often lost in more aggressive compression techniques. Furthermore, the digital twin serves as a safety layer, allowing operators to monitor and adjust actions through visual feedback.
Conclusion
The YOLOv5-ECA system provides a robust technical solution for HMI in complex processing environments, improving operational efficiency and safety. While highly effective, future work will address environmental sensitivities like extreme lighting or occlusions and expand the system to recognize more dynamic and complex industrial gestures.
Summary updated