Consistency and Accuracy of CelebA Attribute Values

Haiyu Wu, Grace Bezold, Manuel Günther, Terrance E. Boult, Michael C. King, Kevin W. Bowyer

TL;DR

This paper reveals significant inconsistencies and errors in the widely-used CelebA facial attribute dataset, showing that many attributes are subjective and poorly labeled. By developing a cleaning workflow for the "mouth slightly open" (MSO) attribute, they demonstrate that using corrected data leads to substantially improved model accuracy and more interpretable results.

What Problem Does This Paper Address?

Facial attribute classification is a crucial task in computer vision, with applications ranging from face recognition to generating synthetic images. The CelebA dataset, with its 40 binary attributes, is a cornerstone for research in this area. However, the paper highlights a critical issue: the quality and consistency of the annotations within CelebA are largely unexamined. This lack of rigorous validation means that models trained on this data might be learning from noisy or even contradictory information, potentially hindering progress and leading to models that don't accurately reflect the intended concepts. The authors aim to systematically analyze these annotation issues and propose a method for improving data quality.

Methods

The researchers employ a multi-pronged approach:

  1. Manual Annotation Consistency: They have two annotators independently label 1,000 CelebA images across 40 attributes to measure inter-annotator agreement. This helps identify which attributes are inherently subjective and difficult to label consistently.
  2. Duplicate Image Analysis: They leverage 5,068 duplicate face image pairs within CelebA to assess the consistency of existing attribute labels. If an attribute is consistently labeled, duplicate images should have the same label.
  3. Manual Audit and Error Estimation: For a subset of attributes (specifically the 12 deemed most consistently labelable), they perform a manual audit on CelebA's existing labels to estimate error rates for both "true" and "false" values.
  4. Semi-Automated Cleaning Workflow: For the "mouth slightly open" (MSO) attribute, they develop a novel workflow that combines manual labeling of a small subset with automated classification and agreement-based auditing to clean a larger portion of the dataset. This workflow iteratively refines labels and identifies unusable images or those requiring an "information not visible" tag.
  5. Model Training and Comparison: They train four different deep learning models (AFFACT, MOON, ResNet, DenseNet) using both original and cleaned MSO attribute values to quantitatively and qualitatively assess the impact of data quality on model performance.

Datasets

  • CelebA: The primary dataset used throughout the study. The authors analyze its existing attribute annotations and also create a cleaned version of the "mouth slightly open" (MSO) attribute.
  • Newly Introduced Benchmark: The paper makes available a corrected version of the MSO attribute for CelebA, serving as a new, higher-quality benchmark for this specific attribute.

Key Insights and Ideas

  • Many Attributes are Inherently Subjective: The study reveals that a significant number of commonly used facial attributes (like "high cheekbones," "pointed nose," and "oval face") have very low inter-annotator consistency, suggesting they are too subjective for reliable binary classification.
  • CelebA's Labels are Surprisingly Noisy: Even for attributes that can be labeled consistently, the existing CelebA annotations contain substantial errors. The paper quantifies this, showing error rates as high as 40% for certain attributes and values.
  • Asymmetric Error Rates Matter: The paper highlights that error rates can be highly asymmetric (e.g., very different error rates for "MSO=true" vs. "MSO=false"). This asymmetry can lead models to learn biased representations.
  • Data Cleaning is Crucial for Performance: The most impactful finding is that cleaning noisy labels, even for a single attribute like MSO, leads to demonstrably better and more focused models. This underscores the importance of data quality over sheer dataset size.
  • Visualizing Model Focus: The use of Score-CAM heatmaps provides compelling visual evidence that models trained on cleaned data are more accurately focused on the relevant facial regions (e.g., the mouth for MSO).

Key Results

  • Annotation Consistency: Only 12 out of 40 common attributes achieve ≥ 95% consistency between annotators. Three attributes exhibit near-random consistency (around 50%).
  • Label Inconsistency in CelebA: Across 5,068 duplicate image pairs in CelebA, attributes show contradicting values on 10 to 860 pairs.
  • MSO Error Rates: The estimated error rate for "MSO=false" is around 20.9%, while for "MSO=true" it's about 1.6%.
  • Model Performance Improvement: Using the cleaned MSO attribute, models achieved higher accuracy than state-of-the-art results obtained with original CelebA data. For example, the MOON model reached 95.49% accuracy with cleaned data, compared to 94.16% with original data.
  • Data Cleaning Impact: Mixed training/validation/testing sets with both original and cleaned data resulted in a significant drop in accuracy, emphasizing the need for consistent, high-quality data across the entire pipeline.

Limitations and Future Work

  • Subjectivity of Attributes: The paper acknowledges that some attributes are inherently too subjective to be reliably labeled, suggesting a need to focus on more objective attributes or develop new ways to represent subjective information.
  • Limited Scope of Cleaning: While the MSO attribute was thoroughly cleaned, the study only performed a deep audit on a subset of attributes. The authors imply that a broader cleaning effort across CelebA would be beneficial.
  • "Information Not Visible" Tag: The introduction of an "information not visible" tag for attributes is a practical solution for occluded or unclear features, but its systematic application across all attributes would require further work.
  • Future Directions: The authors suggest that future research datasets should include descriptions of annotation consistency and accuracy. They also propose that for multi-attribute classification, allowing for different subsets of attributes to be used from different images could better handle occlusions. The availability of the cleaned MSO attribute is intended to spur further research in this direction.

Summary updated

Abstract

We report the first analysis of the experimental foundations of facial attribute classification. An experiment with two annotators independently assigning values shows that only 12 of 40 commonly-used attributes are assigned values with >= 95% consistency, and that three (high cheekbones, pointed nose, oval face) have random consistency (50%). These results show that the binary face attributes currently used in this research area could re-focused to be more objective. We identify 5,068 duplicate face appearances in CelebA, the most widely used dataset in this research area, and find that individual attributes have contradicting values on from 10 to 860 of 5,068 duplicates. Manual audit of a subset of CelebA estimates error rates as high as 40% for (no beard=false), even though the labeling consistency experiment indicates that no beard could be assigned with >= 95% consistency. Selecting the mouth slightly open (MSO) attribute for deeper analysis, we estimate the error rate for (MSO=true) at about 20% and for (MSO=false) at about 2%. We create a corrected version of the MSO attribute values, and compare classification models created using the original versus corrected values. The corrected values enable a model that achieves higher accuracy than has been previously reported for MSO. Also, ScoreCAM visualizations show that the model created using the corrected attribute values is in fact more focused on the mouth region of the face. These results show that the error rate in the current CelebA attribute values should be reduced in order to enable learning of better models. The corrected attribute values for CelebA's MSO and the CelebA facial hair attributes will be made available upon publication.
BibTeX
@inproceedings{wu2023consistency,
  author = {Haiyu Wu and Grace Bezold and Manuel Günther and Terrance E. Boult and Michael C. King and Kevin W. Bowyer},
  title = {{Consistency and Accuracy of CelebA Attribute Values}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  doi = {10.1109/cvprw59228.2023.00328},
}
Figure