Measuring Representation of Race, Gender, and Age in Children's Books: Face Detection and Feature Classification in Illustrated Images

Teodora Szasz, Emileigh Harrison, Ping-Jung Liu, Ping-Chang Lin, Hakizumwami Birali Runesha, Anjali AdukiaWACV 2022

TL;DR

This paper introduces novel AI tools to automatically analyze the representation of race, gender, and age in children's books, addressing the limitations of existing methods that struggle with illustrations. By creating new datasets and training specialized models, they provide a scalable way to understand how diverse identities are portrayed in this influential medium.

What Problem Does This Paper Address?

Children's books are powerful tools for shaping young minds, conveying messages about society and the roles individuals play within it. The visual content, particularly illustrations, plays a crucial role in this process, especially for pre-literate children. However, understanding the representation of different identities (race, gender, age) in these books has traditionally relied on time-consuming manual content analysis, limiting the scope and scale of such studies. Existing AI-powered face detection and classification models are primarily trained on photographs and perform poorly on illustrations, which constitute over 80% of children's book imagery. These illustrations often feature diverse artistic styles, non-human characters, and anthropomorphic features, posing a significant challenge for standard computer vision techniques. This gap prevents a systematic, large-scale assessment of how diverse identities are depicted, potentially perpetuating biases.

Methods

The authors developed a two-pronged approach:

  1. Illustrated Face Detection: They tackled the challenge of detecting faces in illustrations by:
  2. Curating a new dataset (IllusFace 1.0): This dataset contains 5,403 manually labeled illustrated faces from children's books, covering a wide range of drawing styles.
  3. Training AutoML-based models: They trained two models, FDAI (using IllusFace 1.0) and FDAI_iC (using the iCartoon dataset), leveraging Google's AutoML Vision platform. This platform automates the process of designing and fine-tuning neural network architectures.
  4. Feature Classification (Race, Gender, Age): To classify the attributes of detected faces, they:
  5. Curated a new dataset (CBFeatures 1.0): This dataset includes 980 manually labeled faces from children's books, annotated for gender, age, and race by three different raters, importantly including an "unsure" label to account for ambiguity.
  6. Trained AutoML-based models: They trained a model named FCA using CBFeatures 1.0. They also trained FCA_UTK using the publicly available UTKFace dataset (trained on photographs) for comparison.
  7. Benchmarked against an existing model: They compared their FCA model against FairFace (FCF), a well-established model for facial attribute classification.

Datasets

  • IllusFace 1.0 (New): A custom dataset of 5,403 illustrated faces from children's books, manually labeled with bounding boxes. This dataset is crucial for training models to detect faces in illustrations.
  • iCartoon (Publicly Available): Used to train one of their face detection models (FDAI_iC).
  • CBFeatures 1.0 (New): A dataset of 980 manually labeled faces from children's books, annotated for gender, age, and race by three raters, including an "unsure" category. This dataset is used to train their primary feature classification model (FCA).
  • UTKFace (Publicly Available): A large dataset of over 20,000 photographs used to train a comparative feature classification model (FCA_UTK).
  • FairFace (Publicly Available): Used as a benchmark for their feature classification models (FCF).

Key Insights and Ideas

  • Illustrations are a distinct challenge: Standard face detection models trained on photographs perform poorly on illustrated content, highlighting the need for specialized datasets and models.
  • AutoML is effective for specialized tasks: Google's AutoML Vision platform proved valuable for efficiently training models tailored to illustrated faces and their attributes, reducing the need for extensive manual hyperparameter tuning.
  • Subjectivity in manual labeling: The authors acknowledge that manual labeling of illustrated faces, especially for attributes like race and age, can be subjective and lead to inter-rater variability, impacting the "ground truth" for evaluation.
  • "Unsure" labels are important: Including an "unsure" category in their CBFeatures 1.0 dataset is a practical step to account for the inherent ambiguity in illustrated faces, which can be more pronounced than in photographs.
  • Representation trends over time: The analysis of detected faces across decades reveals increasing numbers of faces in books, suggesting a potential shift in representation, though the specific trends for different demographics require careful interpretation due to model biases.

Key Results

  • Improved Face Detection: Their FDAI model detected 2.5 times more faces in children's books compared to Google Vision API (FDGV), and 1.25 times more than the iCartoon-trained model (FDAI_iC).
  • Higher Precision: FDAI achieved 93.4% precision and 76.8% recall, with an AuPRC of 84% at an IoU of 0.5.
  • Enhanced Race Classification: Their FCA model was 34% more accurate in race predictions compared to the established FairFace (FCF) model.
  • Age Classification Improvement: While performance was moderate with five age labels, regrouping into "children" and "adults" significantly improved accuracy, with FCA achieving 72% accuracy.
  • Gender Classification Trends: FCA and FCA_UTK showed comparable gender distributions, while FCF tended to predict more male faces.
  • Race Distribution Skew: All models showed a tendency to label faces as "White," followed by "Asian" and "Black," with "Latinx + Others" being the least frequently assigned label.

Limitations and Future Work

  • Limitations:
    • Inter-rater reliability: The subjectivity of manual labeling for illustrated faces introduces challenges in establishing a definitive "ground truth."
    • Model biases: While efforts were made to create balanced datasets, inherent biases in the data and models can still influence predictions. The authors note that gender labels are limited to binary classifications.
  • Face-only analysis: The current models focus on faces, not the entire character, which might miss broader representational nuances.
  • Future Work:
    • Expand detection to full characters: Develop models that can detect and analyze entire illustrated characters, not just faces.
  • Interpretability: Employ methods like gradient class activation maps (CAMs) to understand why models make certain predictions, enhancing transparency and trust.
  • Broader identity dimensions: Extend the analysis to include other identity dimensions and intersectionalities.
  • Actionable insights for decision-makers: Continue to refine tools to help educators, caregivers, and publishers make informed decisions about representation in children's content.

Summary updated

BibTeX
@inproceedings{szasz2022measuring,
  author = {Teodora Szasz and Emileigh Harrison and Ping-Jung Liu and Ping-Chang Lin and Hakizumwami Birali Runesha and Anjali Adukia},
  title = {{Measuring Representation of Race, Gender, and Age in Children's Books: Face Detection and Feature Classification in Illustrated Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure