Data-Centric AI Paradigm Based on Application-Driven Fine-grained Dataset Design

Huan Hu, Yajie Cui, Zhaoxiang Liu, Shiguo LianarXiv 2022 · DOI · arXiv

Introduction

In the transition from academic research to industrial deployment, AI models often face a significant performance gap. While a model might achieve high accuracy on standard benchmarks like COCO or Pascal VOC, it frequently suffers from high False Alarm Rates (FAR) in real-world environments. This paper proposes a shift toward a "data-centric" AI paradigm, moving away from traditional coarse binary labeling toward a fine-grained system that explicitly accounts for uncertainty. By introducing an "uncertainty" category for ambiguous samples, the researchers demonstrate a dramatic reduction in false alarms for industrial mask-detection systems, dropping the FAR from 8.9% to just 0.8%.

The Problem of Coarse Labeling

In industrial safety applications, such as monitoring mask-wearing in factories or hospitals, False Alarms (FA) are more than just a nuisance; they are a critical failure point. Frequent incorrect alerts lead to the "crying wolf" effect, where human operators eventually ignore or disable the system entirely.

The authors argue that the root cause of these false alarms is often "coarse" dataset design rather than architectural weaknesses in the neural networks. Standard datasets force data into binary categories (e.g., "Mask" vs. "No Mask"). However, real-world visual data exists on a spectrum of difficulty.

The spectrum of ambiguity between positive and negative samples in real-world data.

As shown in the figure above, many images fall into a "messy middle ground" where a definitive classification is difficult even for humans. A mask might be hanging off an ear, the image might be motion-blurred, or a hand might be covering a mouth. When a model is forced to categorize these ambiguous samples as either strictly "Positive" or "Negative" during training, it introduces "gradient noise," confusing the network and leading to high false alarm rates in production.

Proposed Methodology: Fine-Grained Dataset Design

To address this, the researchers advocate for a Fine-Grained Dataset Design paradigm. Instead of a simple binary classifier, they propose a framework that identifies and isolates "uncertain" samples into three distinct sub-categories:

  1. Positive Samples: Clear, unambiguous cases of the target behavior (e.g., someone definitely not wearing a mask).
  2. Negative Samples: Clear, unambiguous cases of the non-target behavior (e.g., someone wearing a mask correctly).
  3. Uncertainty Samples: These are the isolated "messy" cases, further categorized as:
  4. Irregularly Wearing (IW): Masks that are present but do not cover the nose or mouth.
  5. Low Quality (LQ): Images compromised by poor lighting, extreme angles, or heavy motion blur.
  6. Mask-like Occlusion (MLO): Objects like hands, phones, or tools that mimic the visual appearance of a mask.
Evolution from binary classification (a) to a fine-grained uncertainty-aware dataset (d).

By training the model with an explicit "uncertainty" class, the network learns to distinguish between a clear violation and a situation where the data quality is insufficient to make a definitive call. This allows the system to prioritize precision, ensuring that the "Positive" detections are highly reliable.

Experimental Results

The researchers collected a specialized dataset from 75 real-world scenarios, including hospitals, schools, and factories, totaling over 10,000 head-crop images. They conducted ablation studies using a ResNet50 backbone to measure how extracting different "uncertain" categories affected the False Alarm Rate (FAR).

Dataset Design False Alarm Rate (FAR)
Original (Binary) 8.9%
Extracting Low Quality (LQ) only 3.8%
Extracting Irregularly Wearing (IW) only 4.1%
Extracting All (IW + LQ + MLO) 0.8%

The results indicate that Low Quality (LQ) images were the single largest contributor to false alarms. However, the most significant improvement came from isolating all three types of uncertainty, resulting in a 91% reduction in false alarms compared to the standard binary training method.

Conclusion and Key Insights

The core insight of this research is that giving an AI model the option to say "I don't know" (via the uncertainty class) significantly improves its performance on clear-cut cases. This approach aligns with the broader "Data-Centric AI" movement, demonstrating that a 10x improvement in error rates can often be achieved more effectively through refined labeling logic than through complex architectural tweaks. While fine-grained labeling is more labor-intensive, the resulting gains in system reliability and user trust are indispensable for industrial-grade AI applications.

Summary updated

Abstract

Deep learning has a wide range of applications in industrial scenario, but reducing false alarm (FA) remains a major difficulty. Optimizing network architecture or network parameters is used to tackle this challenge in academic circles, while ignoring the essential characteristics of data in application scenarios, which often results in increased FA in new scenarios. In this paper, we propose a novel paradigm for fine-grained design of datasets, driven by industrial applications. We flexibly select positive and negative sample sets according to the essential features of the data and application requirements, and add the remaining samples to the training set as uncertainty classes. We collect more than 10,000 mask-wearing recognition samples covering various application scenarios as our experimental data. Compared with the traditional data design methods, our method achieves better results and effectively reduces FA. We make all contributions available to the research community for broader use. The contributions will be available at https://github.com/huh30/OpenDatasets.
BibTeX
@misc{hu2022data,
  author = {Huan Hu and Yajie Cui and Zhaoxiang Liu and Shiguo Lian},
  title = {{Data-Centric AI Paradigm Based on Application-Driven Fine-grained Dataset Design}},
  year = {2022},
  eprint = {2209.09449},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.09449},
}
Figure