Abstract
Current emotion AI models often struggle to accurately interpret the affective states of older adults, particularly those from non-English-speaking backgrounds. This study evaluates state-of-the-art affective computing tools on Spanish-speaking seniors using a multi-modal approach. By introducing a new human-to-human interview dataset, the research reveals that generalized models frequently fail to capture the nuanced, context-dependent, and highly individual ways the elderly express emotion.
Introduction
Most affective computing models—the underlying technology for virtual assistants and social robots—are trained on datasets dominated by younger, English-speaking populations. This demographic gap leads to significant misinterpretations when these models are deployed to support the well-being of seniors. Older adults exhibit unique behavioral patterns influenced by age-related physical changes and cultural nuances. For instance, a "pensive" expression might be misclassified as "sadness," or a social smile might be mistaken for genuine happiness. This research aims to quantify these discrepancies and provide a realistic benchmark for Spanish-speaking older adults.
The study utilizes two distinct datasets to compare interaction contexts:
- The Wizard of Oz (WoZ) Dataset: Spanish-speaking seniors interacting with a simulated virtual avatar in a controlled environment.
- The Short Interviews Dataset: A new dataset featuring natural, semi-structured human-to-human interviews with 16 Spanish-speaking participants (aged 62+).
Methods
The researchers developed a multi-modal analysis pipeline to process video and audio data, as illustrated in Figure 1.
The pipeline includes:
- Smile Detection: A Transformer-based video architecture (MARLIN) fine-tuned to distinguish smiles from speech-related mouth movements.
- Facial Expression Recognition (FER): EfficientNet and MobileNet models pre-trained on AffectNet to classify eight core emotions.
- Speech & Sentiment Analysis: Audio transcription via OpenAI’s Whisper and sentiment categorization (Positive, Negative, Neutral) using RoBERTuito, a model specialized in Spanish social media text.
- Temporal Alignment: A custom oversampling/downsampling strategy to sync frame-by-frame facial expressions with utterance-level sentiment.
Results and Analysis
The evaluation revealed significant misalignments between human annotations and model predictions, particularly in facial expression recognition.
Facial Expression and Sentiment Misalignment
While models performed well on "Happiness," they struggled with other categories. As shown in Figure 2, human-annotated "Neutral" expressions were frequently misclassified as "Sadness" (40.2%) by the FER models. Furthermore, a "Subjectivity Gap" was evident: human labels like "pensive" were forced into discrete categories like "Sadness" (33.1%) or "Neutral" by the models, highlighting the limitations of standard discrete emotion classifications.
The "Social Smile" and Context
A key insight was the prevalence of the "social smile." In this demographic, smiles often co-occur with "Neutral" speech sentiment, serving as a sign of politeness or active listening rather than internal happiness. Additionally, context significantly influenced model performance. The human-agent (WoZ) dataset skewed toward positive/neutral outputs, while the human-human interviews captured a wider, more authentic range of emotional responses, including more negative sentiment.
Individual Variability
Per-user analysis showed extreme variability in expressiveness. Sentiment prediction accuracy ranged from 0.25 to 0.80 across different users, underscoring that "one-size-fits-all" models are insufficient for this population.
Conclusion
This research highlights the critical need for personalized, context-aware affective models. Current generalized models fail to account for the unique emotional expressions of Spanish-speaking older adults. Future work should focus on learning individual "baselines" of expression and expanding datasets to include more diverse, non-English-speaking populations to ensure inclusive assistive technology.
Summary updated