SuPEr-SAM: Using the Supervision Signal From a Pose Estimator to Train a Spatial Attention Module for Personal Protective Equipment Recognition

Adrian Sandru, Georgian-Emilian Duta, Mariana-Iuliana Georgescu, Radu Tudor IonescuWACV 2021

TL;DR

This paper introduces SuPEr-SAM, a novel deep learning approach for automatically detecting Personal Protective Equipment (PPE) in images. By leveraging a pose estimator during training to guide a spatial attention module, SuPEr-SAM significantly improves PPE recognition accuracy without increasing inference time, making it a practical solution for workplace safety monitoring.

What Problem Does This Paper Address?

Ensuring workers wear appropriate Personal Protective Equipment (PPE) like helmets and masks is crucial for safety in hazardous environments and for public health compliance (e.g., during pandemics). However, manual monitoring is impractical, and existing automated systems often struggle with accuracy, especially for small objects or in challenging conditions. Traditional deep learning methods for PPE detection typically involve training object detectors directly or using a two-stage approach of person detection followed by PPE classification. While the latter is generally more accurate, this paper aims to further enhance its performance by introducing a more sophisticated way to train the classification component.

Methods

The core of the proposed method, SuPEr-SAM, is a two-stage pipeline:

  1. Person Detection: A pre-trained YOLOv3 object detector is used to identify people in the input images.
  2. PPE Classification with Spatial Attention: For each detected person, a cropped image region is fed into a classifier. The novelty lies in how this classifier is trained:
  3. Spatial Attention Module (SAM): A SAM is integrated into the classifier (specifically, a MobileNetV2 architecture). This module learns to focus on relevant parts of the image for classification.
  4. Pose Estimator Supervision (SuPEr): Crucially, during training only, an articulated body pose estimator is used. This estimator provides a "pseudo-ground-truth" attention map, indicating where specific body parts (relevant to the PPE being detected, e.g., the head for helmets) are located. This supervision signal is used to train the SAM.
  5. Joint Loss Function: The classifier is trained to minimize a combined loss: a standard classification loss (binary cross-entropy) and an attention loss, which penalizes deviations from the pose-estimator-derived attention map.

During inference, the pose estimator is removed, meaning the computational overhead of the attention module is minimal.

Datasets

The paper evaluates their approach on three distinct datasets, each focusing on a specific type of PPE:

  • Helmets Data Set: Contains images of people wearing and not wearing helmets.
  • Boots Data Set: Features images of people wearing and not wearing protective boots.
  • Surgical Masks Data Set: Includes images of people with and without surgical masks.

The authors collected these datasets and split them into training, validation, and testing sets using stratified sampling to maintain class ratios.

Key Insights and Ideas

  • Pose as a "Teacher" for Attention: The most significant insight is using a pose estimator not for direct detection, but as a supervisory signal to train a spatial attention mechanism. This allows the attention module to learn where to look for PPE more effectively than it might learn on its own.
  • Attention Module Benefits from Supervision: Standard spatial attention modules, when integrated without specific training guidance, can sometimes even degrade performance (as seen in their MobileNet+SAM baseline). This highlights that the way attention is trained is critical.
  • Minimal Inference Overhead: The clever design of using the pose estimator only during training means that the final deployed system (SuPEr-SAM) incurs almost no additional computational cost compared to a classifier with a standard SAM.
  • Two-Stage Superiority: The results reinforce the established understanding that two-stage approaches (detect person, then classify PPE) generally outperform single-stage object detectors for PPE recognition, especially for smaller items.

Key Results

  • Significant Accuracy Gains: SuPEr-SAM consistently outperforms baseline methods (standard MobileNet and MobileNet with a standard SAM) across all three PPE datasets (Helmets, Boots, Surgical Masks) and various metrics (accuracy, precision, recall, F1-score). For example, on the Helmets dataset, SuPEr-SAM achieved a 98.92% accuracy, a notable improvement over the baseline's 96.90%.
  • Improved Stability: The proposed method exhibits lower standard deviations across multiple runs, indicating greater robustness and stability compared to the baselines.
  • Competitive Inference Speed: The complete SuPEr-SAM pipeline achieves a processing speed of approximately 32.6 ms per image (around 30 FPS), making it suitable for real-time applications. The addition of the SAM adds only about 0.6 ms to the classifier's inference time.
  • Qualitative Improvements: Visualizations show that the attention maps generated by SuPEr-SAM are much more correlated with the actual locations of PPE items (as indicated by the pseudo-ground-truth from the pose estimator) compared to the standard SAM, demonstrating that the model is indeed focusing on the correct regions.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the standard SAM baseline sometimes performed worse than the base MobileNet, suggesting that integrating attention without proper supervision can be detrimental.
  • The performance on the Surgical Masks dataset showed that while SuPEr-SAM improved precision, the macro-averaged recall was slightly lower than MobileNet+SAM, though the overall F1-score was still superior.
  • Future Work:
    • The authors plan to extend their work beyond PPE recognition, integrating SuPEr-SAM into other neural architectures and potentially for other computer vision tasks.
  • They aim to further explore the integration of SuPEr-SAM in different neural network architectures.

Summary updated

BibTeX
@inproceedings{sandru2021super,
  author = {Adrian Sandru and Georgian-Emilian Duta and Mariana-Iuliana Georgescu and Radu Tudor Ionescu},
  title = {{SuPEr-SAM: Using the Supervision Signal From a Pose Estimator to Train a Spatial Attention Module for Personal Protective Equipment Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure