FaceMask: A New Image Dataset for the Automated Identification of People Wearing Masks in the Wild

Michalis Vrigkas, Evangelia-Andriana Kourfalidou, Marina E. Plissiti, Christophoros NikouSensors 2022 · OpenReview

TL;DR

This paper introduces "FaceMask," a new, publicly available dataset of over 4,800 images featuring people wearing or not wearing masks in diverse real-world scenarios. They demonstrate its utility by training and evaluating YOLOv3, YOLOv4, and YOLOv4-tiny models, finding YOLOv4 to be the most performant.

What Problem Does This Paper Address?

The COVID-19 pandemic made mask-wearing a necessity, highlighting the need for automated systems to detect individuals wearing masks, especially in public spaces for health monitoring. While deep learning excels at object detection, a significant hurdle has been the lack of comprehensive, publicly available datasets of people wearing masks in varied, "in-the-wild" conditions. Existing datasets are often limited in size, diversity, or realism, hindering the development and evaluation of robust mask detection algorithms. This paper aims to bridge this gap by creating and releasing the FaceMask dataset.

Methods

The core of this work is the creation of the FaceMask dataset. The authors collected approximately 4,866 images by searching Google Images using keywords related to mask-wearing and crowds. These images were then meticulously processed:

  • Data Collection: Extensive web scraping using the SERP API.
  • Duplicate Removal: Employed AntiDupl.NET to eliminate redundant images.
  • Annotation: Utilized the LabelImg tool to annotate images with bounding boxes and class labels ("Mask" or "No_Mask"). A two-observer system was used to ensure annotation accuracy, with disagreements resolved by a third observer.
  • Model Training and Evaluation: The researchers trained and evaluated three versions of the YOLO (You Only Look Once) object detection network: YOLOv3, YOLOv4, and YOLOv4-tiny. These models were trained using the Darknet framework on a powerful GPU-equipped workstation. Performance was assessed using metrics like Mean Average Precision (mAP), Average IoU, and class-specific AP for "Mask" and "No_Mask" categories. They also performed a k-fold cross-validation and a two-fold cross-validation to assess robustness.

Datasets

  • FaceMask Dataset: The primary contribution. It comprises 4,866 annotated images, categorized into:
  • Individual faces (3,529)

    • Partially occluded faces (1,307)
    • Crowded scenes (more than 10 faces) (994)
    • Blurred faces (1,068) The dataset is made publicly available via a GitHub repository.
  • Existing Datasets Mentioned: The paper references other datasets like MegaFace (3.3M facial images) and MS-Celeb-1M (10M images) for context on large-scale face recognition datasets, and also mentions datasets used in specific mask detection challenges (e.g., MFR challenge, Zhang et al. [21], Ding et al. [30]).

Key Insights and Ideas

  • The "In-the-Wild" Challenge: The paper strongly emphasizes that real-world mask detection requires datasets that capture the messiness of reality – diverse lighting, poses, occlusions, and crowded scenes. Simply having images of people with masks isn't enough; they need to be representative of how people actually appear in public.
  • YOLOv4's Superiority: The experiments clearly indicate that YOLOv4, with its advanced features like CSPDarknet53 and Bag-of-Freebies/Specials techniques, significantly outperforms YOLOv3 and the lighter YOLOv4-tiny in terms of accuracy (mAP).
  • Trade-offs in YOLOv4-Tiny: While YOLOv4-tiny offers faster inference due to its simplified architecture, this comes at the cost of reduced accuracy compared to its larger counterpart.
  • Importance of Cross-Validation: The two-fold cross-validation results showed improved performance, suggesting that robust evaluation requires careful data partitioning to avoid overfitting and to better estimate real-world performance.
  • Failure Cases Highlight Nuances: The analysis of failure cases reveals that misclassifications often occur with non-standard masks, incorrect mask usage, low-resolution images, or faces at the image border. This points to the need for more nuanced annotation and potentially more sophisticated model architectures.

Key Results

  • The FaceMask dataset contains 4,866 images, split into training (17,497 faces), validation (5,404 faces), and test (4,780 faces) sets.
  • YOLOv4 achieved the highest performance, with a best mAP of 90.04% and an AP for Mask of 89.98%.
  • YOLOv3 achieved a best mAP of 84.95% and an AP for Mask of 83.85%.
  • YOLOv4-tiny achieved a best mAP of 82.56% and an AP for Mask of 81.58%.
  • The two-fold cross-validation showed improved mAP scores across all models compared to initial training, with YOLOv3 showing the largest gain (1.66%).

Limitations and Future Work

  • Dataset Limitations:
    • The paper acknowledges that incorrect ground truth images might exist and suggests a strict revision.
  • Images with irregular illumination and motion blur could lead to misclassifications and suggest the need to extend the dataset with such examples.
  • The current dataset might not fully capture the diversity of mask types beyond surgical masks.
  • Future Work:
    • Extend the dataset with images of people from different nationalities to enhance identification performance.
  • Include a wider variety of mask types (e.g., helms, shield masks).
  • Add a category for correctly worn masks to distinguish from incorrect usage.
  • Deploy trained models on smartphones for real-time classification using camera input.
  • Improve models to better handle inter-class similarity and potentially incorporate contextual knowledge for challenging cases.

Summary updated

Abstract

The rapid spread of the COVID-19 pandemic, in early 2020, has radically changed the lives of people. In our daily routine, the use of a face (surgical) mask is necessary, especially in public places, to prevent the spread of this disease. Furthermore, in crowded indoor areas, the automated recognition of people wearing a mask is a requisite for the assurance of public health. In this direction, image processing techniques, in combination with deep learning, provide effective ways to deal with this problem. However, it is a common phenomenon that well-established datasets containing images of people wearing masks are not publicly available. To overcome this obstacle and to assist the research progress in this field, we present a publicly available annotated image database containing images of people with and without a mask on their faces, in different environments and situations. Moreover, we tested the performance of deep learning detectors in images and videos on this dataset. The training and the evaluation were performed on different versions of the YOLO network using Darknet, which is a state-of-the-art real-time object detection system. Finally, different experiments and evaluations were carried out for each version of YOLO, and the results for each detector are presented.
BibTeX
@inproceedings{vrigkas2022facemask,
  author = {Michalis Vrigkas and Evangelia-Andriana Kourfalidou and Marina E. Plissiti and Christophoros Nikou},
  title = {{FaceMask: A New Image Dataset for the Automated Identification of People Wearing Masks in the Wild}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=XwHvdi3Awo},
}
Figure