Abstract: Missing diversity, equity, and inclusion elements in affective computing datasets directly affect the accuracy and fairness of emotion recognition algorithms across different groups. A literature review reveals how affective computing systems may work differently for different groups due to, for instance, mental health conditions impacting facial expressions and speech or age-related changes in facial appearance and health. Our work analyzes existing affective computing datasets and highlights a disconcerting lack of diversity in current affective computing datasets regarding race, sex/gender, age, and (mental) health representation. By emphasizing the need for more inclusive sampling strategies and standardized documentation of demographic factors in datasets, this paper provides recommendations and calls for greater attention to inclusivity and consideration of societal consequences in affective computing research to promote ethical and accurate outcomes in this emerging field.
Introduction
Affective computing (AC)—the field of artificial intelligence dedicated to recognizing, interpreting, and responding to human emotions—is rapidly transitioning from controlled laboratory settings into high-stakes real-world environments. Today, AC systems are deployed in healthcare for diagnosing depression, in education for monitoring student engagement, and even in border control for security screenings. However, the efficacy of these systems relies heavily on their "data foundations."
Human emotional expression is not a universal constant; it is deeply mediated by age, culture, gender, and mental health. If the datasets used to train these models do not reflect the diversity of the human experience, the resulting technologies risk being inaccurate at best and discriminatory at worst. This paper provides a critical audit of 26 of the most widely used datasets in the field to evaluate the current state of diversity, equity, and inclusion (DEI) in affective computing.
Methodology
To assess the "diversity gap," a meta-analysis was conducted on 26 established benchmark datasets in affective computing. These datasets were selected based on their prevalence in recent literature and represent five primary modalities:
- Face: Facial expressions (e.g., JAFFE, CK+, MMI, BP4D).
- Speech: Vocal tones and acoustics (e.g., Emo-DB).
- Body: Posture and gestures (e.g., FABO, EMILYA).
- Physiology: Heart rate, EEG, and skin conductance (e.g., DEAP, SEED, WESAD).
- Multimodal: Combinations of the above streams (e.g., IEMOCAP, RECOLA, MAHNOB-HCI).
For each dataset, we extracted demographic metadata, focusing on subject count, age ranges, sex/gender distribution, racial/ethnic background, and the inclusion of participants with (mental) health conditions.
The Diversity Gap: Key Findings
The "Undergraduate Bias" and Age
Most affective computing datasets rely on "convenience sampling," typically recruiting students from the researchers' own university environments. Consequently, the mean age across nearly all audited datasets is tightly clustered between 20 and 30 years. Only one dataset (4DFAB) included a diverse age range spanning 5 to 75 years. This lack of representation for older adults is particularly concerning given that aging significantly alters facial muscle tone, skin elasticity, and vocal characteristics. Models trained on 20-year-olds are fundamentally ill-equipped for applications in geriatric care or monitoring age-related health conditions.
Racial and Ethnic Underrepresentation
Documentation of race and ethnicity remains a significant barrier in the field; these factors are frequently omitted from research papers entirely. In datasets where racial composition was reported, the distribution was heavily skewed: 62% of subjects were White, while Black and Latino individuals represented only 7.1% and 9.0% of the populations, respectively. For example, the popular CK+ dataset is 81% Euro-American. This "structured missingness" ensures that emotion-recognition models are less accurate for marginalized groups.
Gender and Health Status
While many datasets achieved a near 50/50 split between male and female participants, reporting remains strictly binary. There was zero representation for non-binary or transgender individuals across all 26 datasets.
Furthermore, a "healthy exclusion" bias persists in physiological research. Many datasets explicitly exclude participants with mental health conditions, pregnancy, or even lifestyle factors like heavy smoking. This creates a paradox: the models are often intended to assist in clinical settings (e.g., detecting PTSD or depression), yet they are trained on data that excludes the very emotional signatures of those populations.
Discussion: The Limits of Multimodal Robustness
A common assumption in the field is that multimodal sensing—combining facial data with physiological signals like heart rate—will naturally increase model robustness and mitigate bias. However, our analysis suggests that if each individual data stream is biased, the combined model may simply amplify those inequities rather than correct them. Without diverse underlying data, the complexity of the model cannot compensate for the fundamental lack of representative human experience.
Recommendations and Future Directions
To move toward affective computing that works for everyone, we propose three primary shifts in research practice:
- Standardized Documentation: Mandatory, standardized demographic reporting should be required for all new datasets to ensure transparency regarding who the technology is designed to serve.
- Inclusive Sampling Strategies: Researchers must move beyond university campuses to recruit diverse populations, specifically targeting age groups and health statuses that are currently underrepresented.
- A Clinical Trial Framework: We suggest that affective computing adopt a framework similar to medical clinical trials. In this model, diversity is not an afterthought but a prerequisite for a technology to be considered "safe" and "effective" for public deployment.
Conclusion
The current data foundations of affective computing are built on a narrow slice of humanity. To ensure that emotion-recognition technologies are ethical and accurate, the field must prioritize diversity and inclusion at the data collection stage. Only by accounting for the full spectrum of human expression can we develop AI that truly understands and serves everyone.
Summary updated