This paper provides a critical audit of the "data foundations" of affective computing—the field of AI dedicated to recognizing and responding to human emotions. By analyzing 26 of the most widely used datasets in the field, the authors reveal a systemic lack of diversity that threatens the fairness and accuracy of emotion-recognition technologies in real-world applications.
TL;DR
Affective computing systems are often trained on "WEIRD" (Western, Educated, Industrialized, Rich, and Democratic) data, leading to significant biases against older adults, people with mental health conditions, and racial minorities. This paper audits 26 major datasets and calls for a "clinical trial" approach to AI data to ensure these technologies work for everyone.
What Problem Does This Paper Address?
Affective computing (AC) is moving out of the lab and into high-stakes environments like healthcare (diagnosing depression), education (monitoring student engagement), and even border control. However, human emotional expression is not universal; it is deeply influenced by age, culture, gender, and mental health.
The authors address the "diversity gap": if an AI is trained primarily on the facial expressions and vocal tones of healthy 20-year-old college students, it will likely fail—or worse, misinterpret—the emotions of an elderly person with Parkinson’s or a neurodivergent individual. This "structured missingness" in data doesn't just lead to lower accuracy; it creates a risk of automated discrimination.
Methods
The researchers conducted a meta-analysis of 26 commonly used datasets in affective computing, selected from a recent comprehensive literature review. They evaluated these datasets across five modalities:
- Speech (vocal tones)
- Face (facial expressions)
- Body (posture and gestures)
- Physiology (heart rate, EEG, skin conductance)
- Multimodal (combinations of the above)
For each dataset, they extracted demographic metadata, including the number of subjects, age ranges, sex/gender distribution, racial/ethnic background, and whether participants with (mental) health conditions were included.
Datasets
The paper audits 26 established benchmarks, including:
- Face: JAFFE, Extended Cohn-Kanade (CK+), MMI, Oulu-CASIA, BP4D.
- Physiology: DEAP, SEED, AMIGOS, WESAD.
- Multimodal: IEMOCAP, RECOLA, MAHNOB-HCI.
- Speech/Body: Emo-DB, FABO, EMILYA.
The authors highlight that while these are the "gold standards" for training emotion-recognition models, they are often demographically narrow and poorly documented.
Key Insights and Ideas
- The "Healthy" Exclusion: Many physiological datasets explicitly exclude participants with mental health conditions, pregnancy, or even "heavy smoking." This means the resulting models are fundamentally "blind" to the emotional signatures of the very populations (e.g., those with depression or PTSD) that affective computing is often intended to help.
- The Undergraduate Bias: Most datasets rely on "convenience sampling"—recruiting students from the researchers' own universities. Consequently, the mean age across almost all datasets is between 20 and 30. Because aging changes facial muscle tone and skin elasticity, these models are ill-equipped for geriatric care.
- Documentation as a Barrier: Race and ethnicity are frequently omitted from research papers entirely. When they are reported, they are heavily skewed; for example, the popular CK+ dataset is 81% Euro-American.
- Multimodal is Not a Magic Bullet: While combining data streams (e.g., face + heart rate) is seen as a way to increase robustness, the authors argue that if each individual stream is biased, the combined model may simply amplify those inequities.
Key Results
- Race: In datasets that reported racial composition, 62% of subjects were White. Black and Latino individuals were significantly underrepresented (7.1% and 9.0%, respectively).
- Age: Only one dataset (4DFAB) included a truly diverse age range (5–75 years). Most others were tightly clustered around the 20–30 age bracket.
- Gender: While many datasets achieved a 50/50 male/female split, reporting remains strictly binary. Representation for non-binary or transgender individuals was non-existent across all 26 datasets.
- Health: The vast majority of "general-purpose" datasets do not mention mental health, and those that do often use "healthy" as a prerequisite for participation.
Limitations and Future Work
The authors acknowledge that their study is limited to the 26 datasets identified in a specific review paper and is constrained by what original authors chose to report.
Future Directions: 1. Standardized Documentation: The authors call for mandatory, standardized demographic reporting for all new datasets. 2. Inclusive Sampling: Researchers should move beyond university campuses to recruit diverse populations, particularly those with different health statuses. 3. Clinical Trial Framework: They suggest that affective computing should adopt guidelines similar to clinical trials in medicine, where diversity is a requirement for the technology to be considered "safe" and "effective" for public use.
Summary updated