A Survey on Mobile Affective Computing

Shengkai Zhang, Pan Hui

TL;DR

This survey explores the burgeoning field of Mobile Affective Computing (MAC), which leverages the ubiquitous sensors in smartphones and wearables to unobtrusively understand and respond to human emotions. It highlights how MAC overcomes the limitations of traditional AC by enabling naturalistic, large-scale data collection, paving the way for more intelligent and personalized human-computer interactions.

What Problem Does This Paper Address?

Traditional Affective Computing (AC) research has often been criticized for studying "impermeable emotions" – emotions investigated outside of their natural social contexts. This approach is problematic because emotions are deeply intertwined with social interactions and are often directed at others. Furthermore, traditional AC methods often rely on lab-based, deliberately posed emotional expressions, which differ significantly from spontaneous, real-world emotions. This limits the ecological validity of findings and the development of truly intelligent systems that can proactively interact with users. The advent of smart mobile devices with a rich array of sensors presents a unique opportunity to address these limitations by enabling unobtrusive, continuous, and naturalistic emotion sensing.

Methods

This survey categorizes and discusses various approaches to Mobile Affective Computing (MAC), focusing on the data collection methods and machine learning techniques employed. Key methods include:

  • Signal Collection: Leveraging sensors like cameras (for facial expressions), microphones (for vocal intonation), accelerometers, gyroscopes, GPS, touch sensors, and even keyboard/mouse interactions.
  • Feature Extraction: Deriving meaningful features from raw sensor data, such as keystroke dynamics (timing, pressure), movement patterns, and usage logs.
  • Machine Learning Classifiers: Employing techniques like Decision Trees, Support Vector Machines (SVM), k-Nearest Neighbors (kNN), Bayesian Networks, and Artificial Neural Networks (ANN) to classify emotional states.
  • Unobtrusive Data Collection: Emphasizing methods that collect data without requiring explicit user participation or awareness, such as analyzing smartphone usage patterns or keystroke dynamics.
  • Crowdsourcing: Utilizing platforms like Amazon Mechanical Turk to generate large-scale, annotated affective datasets.

Datasets

The paper discusses the need for and challenges in creating affective datasets. While specific new datasets are not introduced in this survey, it highlights the limitations of existing ones:

  • Traditional AC Datasets: Often based on "artificial" materials of deliberately expressed emotions, collected in lab settings, which differ from spontaneous behavior.
  • MAC Datasets: The paper emphasizes the potential of MAC to create large-scale, spontaneous affect databases efficiently through unobtrusive monitoring of users in their natural environments. Examples include data collected from smartphone usage logs, keystroke dynamics, and sensor readings during daily activities.

Key Insights and Ideas

  • Ubiquitous Sensing for Naturalistic Emotion: Smartphones and wearables, with their integrated sensors, are ideal platforms for unobtrusively collecting real-world emotional data, overcoming the limitations of lab-based studies.
  • Beyond Facial Expressions: While facial expressions are a common cue, MAC explores a broader range of signals, including touch behavior, keystroke dynamics, and device usage patterns, to infer emotions.
  • Emotion as a Component of Intelligence: The paper argues that understanding and responding to emotions is crucial for building truly intelligent systems that can engage in proactive and empathetic human-computer interaction.
  • The "Impermeable Emotion" Problem: The survey directly addresses the criticism that traditional AC studies emotions in isolation, highlighting the need for context-aware and socially situated emotion analysis.
  • Data Scale and Cost: MAC offers a path to overcome the challenges of expensive and time-consuming data collection and labeling in traditional AC research.

Key Results

The paper is a survey, not an experimental study, so it doesn't present its own quantitative results. However, it summarizes findings from various cited works:

  • Keystroke Dynamics: Studies show accuracies ranging from 77.4% to 87.8% for classifying certain emotional states (confidence, hesitance, nervousness, relaxation, sadness, tiredness) using keystroke patterns.
  • Touch Behavior: Research indicates that touch behavior can discriminate between emotional states with accuracies between 69% and 77% for four emotional states, and higher for arousal and valence.
  • Smartphone Usage Logs: Models using six types of usage information (SMS, email, phone call, application usage, web browsing, location) could estimate moods with a satisfactory accuracy of 67.52% for 7 emotion types.
  • Physiological Signals (Webcam): Studies demonstrated the feasibility of measuring physiological parameters like heart rate from webcam video, showing a strong correlation with emotion without user attention.
  • Stress Detection: Increased stress levels were found to significantly influence typing pressure (>83% of participants) and mouse contact (100% of participants).

Limitations and Future Work

Limitations:

  • Ground Truth Establishment: Obtaining reliable ground truth for emotions, especially spontaneous ones, remains a significant challenge. Manual labeling of large datasets is expensive and inefficient.
  • Self-Interference: Designing friendly applications for affective feedback collection can inadvertently influence user affect, making it difficult to evaluate the true impact.
  • Coordination of Affective Parameters: The lack of a robust mechanism to coordinate and integrate multiple affective parameters (e.g., from different sensors) limits comprehensive understanding and response generation.
  • Data Scale and Diversity: While MAC enables larger datasets, ensuring diversity in participants, contexts, and emotional expressions is crucial.
  • Low Performance in Some Studies: Some studies reported comparatively low performance, particularly in recognizing less frequent emotions like anger, fear, and surprise, highlighting the need for more data and refined models.

Future Work:

  • Scalable and Custom Databases: Building large, custom databases that capture common affect features and distinguishable "affect fingerprints" of individuals.
  • Affective Ecosystems: Developing systems that coordinate multiple smart wearable devices for comprehensive affective data fusion and improved detection accuracy.
  • New Affect-Related Features: Investigating novel emotional features arising from changing lifestyles and the increasing use of smart devices and the internet.
  • Interdisciplinary Collaboration: Emphasizing the need for collaboration between computer scientists, psychologists, and social scientists to advance the understanding of human behavior and emotion.
  • Friendly Application Design: Further research into designing affective applications that are comfortable for users and minimize self-interference.

Summary updated

Abstract

This survey presents recent progress on Affective Computing (AC) using mobile devices. AC has been one of the most active research topics for decades. The primary limitation of traditional AC research refers to as impermeable emotions. This criticism is prominent when emotions are investigated outside social contexts. It is problematic because some emotions are directed at other people and arise from interactions with them. The development of smart mobile wearable devices (e.g., Apple Watch, Google Glass, iPhone, Fitbit) enables the wild and natural study for AC in the aspect of computer science. This survey emphasizes the AC study and system using smart wearable devices. Various models, methodologies and systems are discussed in order to examine the state of the art. Finally, we discuss remaining challenges and future works.
BibTeX
@misc{zhang2014survey,
  author = {Shengkai Zhang and Pan Hui},
  title = {{A Survey on Mobile Affective Computing}},
  year = {2014},
  eprint = {1410.1648},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1410.1648},
}
Figure