Towards affective computing that works for everyone

Tessa Verhoef, Eduard Fosch-Villaronga

This paper provides a critical audit of the "data foundations" of affective computing—the field of AI dedicated to recognizing and responding to human emotions. By analyzing 26 of the most widely used datasets in the field, the authors reveal a systemic lack of diversity that threatens the fairness and accuracy of emotion-recognition technologies in real-world applications.

TL;DR

Affective computing systems are often trained on "WEIRD" (Western, Educated, Industrialized, Rich, and Democratic) data, leading to significant biases against older adults, people with mental health conditions, and racial minorities. This paper audits 26 major datasets and calls for a "clinical trial" approach to AI data to ensure these technologies work for everyone.

What Problem Does This Paper Address?

Affective computing (AC) is moving out of the lab and into high-stakes environments like healthcare (diagnosing depression), education (monitoring student engagement), and even border control. However, human emotional expression is not universal; it is deeply influenced by age, culture, gender, and mental health.

The authors address the "diversity gap": if an AI is trained primarily on the facial expressions and vocal tones of healthy 20-year-old college students, it will likely fail—or worse, misinterpret—the emotions of an elderly person with Parkinson’s or a neurodivergent individual. This "structured missingness" in data doesn't just lead to lower accuracy; it creates a risk of automated discrimination.

Methods

The researchers conducted a meta-analysis of 26 commonly used datasets in affective computing, selected from a recent comprehensive literature review. They evaluated these datasets across five modalities:

  • Speech (vocal tones)
  • Face (facial expressions)
  • Body (posture and gestures)
  • Physiology (heart rate, EEG, skin conductance)
  • Multimodal (combinations of the above)

For each dataset, they extracted demographic metadata, including the number of subjects, age ranges, sex/gender distribution, racial/ethnic background, and whether participants with (mental) health conditions were included.

Datasets

The paper audits 26 established benchmarks, including:

  • Face: JAFFE, Extended Cohn-Kanade (CK+), MMI, Oulu-CASIA, BP4D.
  • Physiology: DEAP, SEED, AMIGOS, WESAD.
  • Multimodal: IEMOCAP, RECOLA, MAHNOB-HCI.
  • Speech/Body: Emo-DB, FABO, EMILYA.

The authors highlight that while these are the "gold standards" for training emotion-recognition models, they are often demographically narrow and poorly documented.

Key Insights and Ideas

  • The "Healthy" Exclusion: Many physiological datasets explicitly exclude participants with mental health conditions, pregnancy, or even "heavy smoking." This means the resulting models are fundamentally "blind" to the emotional signatures of the very populations (e.g., those with depression or PTSD) that affective computing is often intended to help.
  • The Undergraduate Bias: Most datasets rely on "convenience sampling"—recruiting students from the researchers' own universities. Consequently, the mean age across almost all datasets is between 20 and 30. Because aging changes facial muscle tone and skin elasticity, these models are ill-equipped for geriatric care.
  • Documentation as a Barrier: Race and ethnicity are frequently omitted from research papers entirely. When they are reported, they are heavily skewed; for example, the popular CK+ dataset is 81% Euro-American.
  • Multimodal is Not a Magic Bullet: While combining data streams (e.g., face + heart rate) is seen as a way to increase robustness, the authors argue that if each individual stream is biased, the combined model may simply amplify those inequities.

Key Results

  • Race: In datasets that reported racial composition, 62% of subjects were White. Black and Latino individuals were significantly underrepresented (7.1% and 9.0%, respectively).
  • Age: Only one dataset (4DFAB) included a truly diverse age range (5–75 years). Most others were tightly clustered around the 20–30 age bracket.
  • Gender: While many datasets achieved a 50/50 male/female split, reporting remains strictly binary. Representation for non-binary or transgender individuals was non-existent across all 26 datasets.
  • Health: The vast majority of "general-purpose" datasets do not mention mental health, and those that do often use "healthy" as a prerequisite for participation.

Limitations and Future Work

The authors acknowledge that their study is limited to the 26 datasets identified in a specific review paper and is constrained by what original authors chose to report.

Future Directions: 1. Standardized Documentation: The authors call for mandatory, standardized demographic reporting for all new datasets. 2. Inclusive Sampling: Researchers should move beyond university campuses to recruit diverse populations, particularly those with different health statuses. 3. Clinical Trial Framework: They suggest that affective computing should adopt guidelines similar to clinical trials in medicine, where diversity is a requirement for the technology to be considered "safe" and "effective" for public use.

Summary updated

Abstract

Missing diversity, equity, and inclusion elements in affective computing datasets directly affect the accuracy and fairness of emotion recognition algorithms across different groups. A literature review reveals how affective computing systems may work differently for different groups due to, for instance, mental health conditions impacting facial expressions and speech or age-related changes in facial appearance and health. Our work analyzes existing affective computing datasets and highlights a disconcerting lack of diversity in current affective computing datasets regarding race, sex/gender, age, and (mental) health representation. By emphasizing the need for more inclusive sampling strategies and standardized documentation of demographic factors in datasets, this paper provides recommendations and calls for greater attention to inclusivity and consideration of societal consequences in affective computing research to promote ethical and accurate outcomes in this emerging field.
BibTeX
@inproceedings{verhoef2023towards,
  author = {Tessa Verhoef and Eduard Fosch-Villaronga},
  title = {{Towards affective computing that works for everyone}},
  booktitle = {ACII},
  year = {2023},
  doi = {10.1109/acii59096.2023.10388169},
}

Loading related content...

Figure