Editorial: Computational Approaches for Human-Human and Human-Robot Social Interactions

Cigdem Beyan, Vittorio Murino, Gentiane Venture, Agnieszka Wykowska

Introduction: The Silent Language of Social Interaction

Human communication is a complex tapestry woven from more than just spoken words. A significant portion of our social exchange occurs through non-verbal channels—gaze, facial expressions, gestures, and vocal nuances—that convey personality, emotion, and intent. While social psychologists have studied these signals for decades, the automation of their analysis remains a significant frontier in computer science. This editorial introduces a curated collection of research focused on using machine learning and computational modeling to decode this "silent language," bridging the gap between social psychology and robotics.

The Dual Challenge: HHI and HRI

The research presented identifies two primary hurdles in the field of social computation. First is the Human-Human Interaction (HHI) Gap: there is a pressing need for robust computational tools to analyze human interactions to better understand social cognition and disorders such as Autism Spectrum Disorder (ASD). Second is the Human-Robot Interaction (HRI) Challenge: for robots to transition from mere tools to social partners, they must possess "social intelligence." This requires robots to not only perceive and interpret human signals but also to emit signals that are legible, trustworthy, and engaging to humans.

Computational Methodologies for Social Decoding

The featured studies employ a diverse array of computational approaches to tackle these challenges:

  • Machine Learning (ML): Extensively used to decode internal states, such as engagement or emotional valence, from video and skeletal data.
  • Motion Energy Analysis (MEA) & Support Vector Machines (SVM): Applied to quantify interpersonal synchrony and classify social difficulties in clinical populations.
  • Reinforcement Learning (RL): Utilized to help robots learn "legible" movement patterns—trajectories that humans can easily predict, fostering a sense of safety.
  • Auto-encoders: Employed for cross-cultural analysis of acoustic synchronization in speech.
  • Computational Modeling: Development of real-time systems that adjust a virtual agent's perceived "warmth" and "competence" to maximize user engagement.

Diverse Datasets and Ecological Validity

The collection highlights several benchmark datasets that push the boundaries of social AI:

  • The Lammers et al. Dataset: A benchmark of everyday actions expressing various emotions, using motion capture rendered into standardized virtual characters to control for physical variables.
  • Cross-Cultural Acoustic Data: A large-scale dataset of speech dyads spanning six cultures, facilitating the study of interpersonal synchronization.
  • "In-the-wild" Field Data: Observations from a Pepper robot acting as a salesperson in a public space, providing crucial ecological validity to HRI theories outside the laboratory.

Key Insights: From Body Pose to Robot Backstories

Several surprising findings emerged from this research collection. One notable insight is that ML models can decode internal states from low-dimensional 2D body pose data almost as effectively as from full-resolution video. This suggests that the structural dynamics of movement are often more informative than fine-grained visual details.

In the realm of HRI, the "power of backstory" was evident. In studies with children, a robot’s "entrainment" (mimicking behavior) combined with a relatable backstory significantly increased rapport and compliance. Furthermore, interpersonal synchrony (IS) was confirmed as a critical component of empathy, with computational tools now able to quantify IS to assist in understanding social challenges in ASD.

Finally, the research emphasizes that robot "legibility"—the ability for a human to predict a robot's next move—is a learnable trait. Through reinforcement learning, robots can adapt their behavior in collaborative tasks to become more predictable and trusted.

Results and Future Horizons

The results across these studies are promising. ML models achieved high performance in classifying interaction valence using skeletal data, and robots in "in-the-wild" settings saw higher customer engagement when they successfully projected social awareness. Cross-culturally, partners tended to synchronize speech acoustics, though specific patterns varied by culture.

However, the automatic realization of social analysis is still in its infancy. Future work must move beyond controlled lab environments to more "in-the-wild" experiments. There is also a need to explore the neural underpinnings of how humans process signals from non-human agents and to integrate multi-modal data (gaze, voice, posture) into a holistic "social sense" for AI.

Summary updated

Abstract

Automatised detection and analysis of nonverbal social signals can be of particular relevance not only to human-human interaction (HHI), but also to human-robot interaction (HRI). Over the last decade, much research effort has been dedicated to improving robots' capabilities regarding perceiving, interacting, and cooperating with humans. Indeed, social HRI requires augmenting robots' standard functionality by the ability to recognize and interpret human social signals, in order to be able to engage naturally and intuitively with a human. Simultaneously, research efforts are being put in examining the human side of HRI, namely, the human mechanisms social cognition in interactions with artificial agents (embodied robots specifically). This is crucial in order to understand how the human brain processes social signals carried out by non-human agents, and whether such agents can evoke mechanisms of social cognition in humans. Also in this case, ML techniques have proved to be useful to explore patterns of neural and behavioural activity of the human counterparts. This Research Topic is dedicated to exploring computational techniques for the analysis of nonverbal social signals in HHI as well as HRI. Specifically, we focus on ML methodologies, as well as other computational approaches for nonverbal behaviour understanding and multi-modal data analysis. It brings together ten selected papers that reflect some of the current computational approaches applied to HHI and HRI.Bartle...
BibTeX
@article{beyan2020editorial,
  author = {Cigdem Beyan and Vittorio Murino and Gentiane Venture and Agnieszka Wykowska},
  title = {{Editorial: Computational Approaches for Human-Human and Human-Robot Social Interactions}},
  journal = {Frontiers Robotics AI},
  year = {2020},
  doi = {10.3389/frobt.2020.00055},
}
Figure