Casual Conversations: A Dataset for Measuring Fairness in AI

Caner Hazirbas, Joanna Bitton, Brian Dolhansky, Jacqueline Pan, Albert Gordo, Cristian Canton-Ferrer

Abstract

Researchers at Meta AI have released Casual Conversations, a dataset comprising over 45,000 videos of 3,011 participants designed specifically to audit AI models for fairness. By utilizing self-identified age and gender alongside expert-labeled skin tones and lighting conditions, this benchmark exposes demographic biases in computer vision models that often go undetected in traditional datasets.

Introduction

Computer vision models for facial analysis—such as age estimation or gender classification—frequently suffer from "algorithmic bias." These models tend to perform significantly better on individuals with lighter skin tones or specific age groups because the datasets used to train them are demographically lopsided. The authors identify three major gaps in existing benchmarks: inaccurate third-party labeling, a lack of subject diversity, and a failure to account for environmental factors like ambient lighting.

Methodology

The team recorded over 45,000 videos of 3,011 paid participants across five U.S. cities. To ensure the dataset could be used to measure fairness accurately, they implemented a rigorous annotation pipeline:

  • Self-Identification: Participants provided their own age and gender, ensuring the "ground truth" reflects how individuals actually identify rather than relying on annotator guesses.
  • Skin Tone Labeling: Eight trained raters used the Fitzpatrick scale (a 6-point scale from "pale white" to "deeply pigmented") to label apparent skin tones.
  • Lighting Control: Videos were recorded in both "Bright" and "Dark" ambient lighting to test model robustness against environmental variability.

Dataset Characteristics

The primary contribution is the Casual Conversations dataset. It is unique among benchmarks like Adience or FairFace because it combines high-volume video data with self-reported labels and explicit lighting annotations.

Distribution of participants across age, gender, skin type, and lighting conditions

As shown in the distribution above, the dataset maintains significant representation across various age groups and skin types, with a specific focus on achieving a more uniform distribution for testing purposes.

Key Results

The authors’ audit of state-of-the-art (SOTA) models—including Levi & Hassner, LMTCNN, and LightFace—revealed stark disparities in performance across demographic buckets.

Performance metrics for gender classification across demographic and environmental categories

Key findings include:

  • The Skin Tone Gap: Gender classification precision for individuals with darker skin tones (Types V & VI) was drastically lower—by more than 20% on average—compared to those with lighter skin tones.
  • Age Bias: Models were generally more successful at classifying the gender of older individuals (45+ years old) than younger ones.
  • Lighting Robustness: Surprisingly, the tested models performed nearly as well on "Dark" videos as they did on "Bright" ones, suggesting that modern architectures handle low light better than they handle diverse skin pigmentation.
  • Model Rankings: While LightFace was the most accurate model overall, it still exhibited the same patterns of demographic bias as the older, less accurate models.

Conclusion and Future Work

Casual Conversations provides a vital tool for researchers to audit their models for more inclusive AI development. The authors acknowledge limitations, such as the simplistic categorization of gender and the geographic scope limited to the United States. They encourage the research community to extend the dataset's annotations for other tasks, such as audio understanding, image inpainting, or temporal consistency in video.

Summary updated

Abstract

This paper introduces a novel fairness dataset to measure the robustness of AI models to a diverse set of age, genders, apparent skin tones and ambient lighting conditions. Our dataset is composed of 3,011 subjects and contains over 45,000 videos, with an average of 15 videos per person. The videos were recorded in multiple U.S. states with a diverse set of adults in various age, gender and apparent skin tone groups. A key feature is that each subject agreed to participate for their likenesses to be used. Additionally, our age and gender annotations are provided by the subjects themselves. A group of trained annotators labeled the subjects' apparent skin tone using the Fitzpatrick skin type scale. Moreover, annotations for videos recorded in low ambient lighting are also provided. As an application to measure robustness of predictions across certain attributes, we evaluate the state-of-the-art apparent age and gender classification methods. Our experiments provides a through analysis on these models in terms of fair treatment of people from various backgrounds.
BibTeX
@inproceedings{hazirbas2021casual,
  author = {Caner Hazirbas and Joanna Bitton and Brian Dolhansky and Jacqueline Pan and Albert Gordo and Cristian Canton-Ferrer},
  title = {{Casual Conversations: A Dataset for Measuring Fairness in AI}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  doi = {10.1109/cvprw53098.2021.00258},
}
Figure