Introduction
While the field of Affective Computing has made significant strides in emotion recognition (detecting what a person is feeling) and emotion generation (expressing emotions), a critical gap remains: the integration of these emotions into meaningful decision-making and behavior. Current AI agents often lack a holistic understanding of social interactions, struggling to interpret how social roles, environmental settings, and cultural norms dictate appropriate behavior. Without this context, AI remains socially clumsy, unable to navigate the nuances of human interaction.
This research program, led by Aarti Malhotra, outlines a multi-disciplinary approach to developing socially intelligent agents. By leveraging Affect Control Theory (ACT), the research moves beyond simple sentiment analysis toward a framework where AI can understand social context and identities to predict "social incoherence" and recommend behaviors that feel authentically human.
Theoretical Framework: Affect Control Theory
The foundation of this work lies in Affect Control Theory (ACT), a sociological theory suggesting that individuals navigate social interactions by attempting to maintain emotional consistency within their perceived social roles. When an interaction deviates from these cultural expectations, it creates "deflection" or social incoherence.
To translate this into a computational model, the research utilizes BayesACT, a probabilistic extension of ACT. BayesACT allows the AI to model the inherent uncertainty and variance in emotional sentiments across different cultures and contexts, making the agent more robust when faced with the complexities of real-world social nuances.
Methodology and Technical Implementation
The research is structured around five key Research Questions (RQs) and employs a diverse technical toolkit:
1. Computational Simulations
To model how perceived social interactions guide behavior, the research utilizes Nengo, a large-scale brain-simulation framework. Additionally, BayesACT-driven bots are implemented in Game Theory scenarios, such as the "Prisoner’s Dilemma," to observe and refine human-agent interaction dynamics.
2. Computer Vision and Deep Learning
A significant portion of the research focuses on the visual perception of social scenes. The author developed a novel spatio-temporal transformer network designed for multi-task learning. Unlike traditional Convolutional Neural Networks (CNNs), these transformers better capture the evolving relationship between individuals and their environment over time, allowing for more accurate social event and group emotion prediction.
3. Inference Networks for Social Incoherence
The research builds an inference network that utilizes pre-trained models to identify place context, identity, and behavior. By comparing observed actions against normative expectations derived from ACT, the system can detect social "incoherence." This allows the AI to either adjust its own behavior or provide recommendations to minimize social conflict.
Data Foundations
The models are trained and validated using several specialized datasets:
- VGAF (Video level Group AFfect): For social event and group emotion prediction.
- TV Human Interactions & MovieGraphs: To validate social incoherence detection and behavior recommendation networks.
- Identity Annotation Surveys: Utilizing datasets such as those by Li et al. (2017) for social relationship recognition.
Key Insights
- Identity as a Social Anchor: Social interaction is not merely about raw emotion; it is dictated by the "social face" or identity a person presents. An AI must recognize whether it is interacting with a "doctor," a "friend," or a "stranger" to determine the appropriate response.
- The Metric of Incoherence: By modeling normative behaviors, the AI gains a mathematical basis for identifying when a situation is "off." This provides a clear trigger for behavioral correction.
- Superiority of Transformers: The transition to spatio-temporal transformers has shown significant performance gains over traditional CNN architectures in capturing the temporal dynamics of social events.
Results and Future Directions
This doctoral research has already yielded several key milestones. The developed spatio-temporal transformer model outperformed state-of-the-art baselines on the VGAF dataset. Furthermore, the work has successfully bridged the gap between sociology and computer science, with findings accepted at major venues including Human-Robot Interaction (HRI) and the International Conference on Cognitive Modeling (ICCM).
Future work will focus on multi-modal integration, incorporating speech and text cues to provide a more comprehensive understanding of social scenes. Additionally, the author aims to refine identity and behavior classifiers for high-stakes domains, such as healthcare (e.g., socially aligned robots for dementia care) and education, where social intelligence is paramount.
Summary updated