The impact of facial expression and communicative gaze of a humanoid robot on individual Sense of Agency

Maria Lombardi, Cecilia Roselli, Kyveli Kompatsiari, Federico Rospo, Lorenzo Natale, Agnieszka Wykowska

TL;DR

This study investigates how a humanoid robot's social cues—facial expressions and gaze—influence a person's sense of agency (SoA), the feeling of control over one's actions. The research found that the robot's gaze significantly impacts SoA, while its facial expressions do not, suggesting gaze is a more potent factor in shaping our perception of control in human-robot interactions.

What Problem Does This Paper Address?

The feeling of "Sense of Agency" (SoA) is fundamental to our experience of being in control of our actions and their outcomes. While SoA has been extensively studied, its modulation by social cues, particularly in the context of human-robot interaction, remains an open question. Specifically, how do a robot's emotional expressions and its gaze affect our implicit sense of agency? Existing research has shown that emotional outcomes can influence SoA, and gaze is a powerful social cue in human interactions, but their combined or individual impact on SoA with a physically embodied agent like a robot is less understood. This paper aims to bridge this gap by systematically examining how a humanoid robot's facial expressions and gaze direction influence participants' implicit SoA using a naturalistic interaction paradigm.

Methods

The researchers employed the temporal interval estimation paradigm to measure implicit SoA. This method relies on the Intentional Binding (IB) effect, where participants tend to perceive the time between their voluntary action and its sensory consequence as shorter than it actually is. A greater temporal compression (shorter perceived interval) indicates a stronger sense of agency.

Participants performed a natural action: turning their head towards a humanoid robot (the iCub). The outcome of this action was manipulated across three experiments:

  • Experiment 1: The robot displayed either a happy or a sad facial expression.
  • Experiment 2: The robot engaged in either mutual gaze (looking at the participant) or averted gaze.
  • Experiment 3: A within-subjects design combined both facial expression (happy/sad) and gaze (mutual/averted) to assess their combined and independent effects.

The iCub robot was equipped with a gaze detection system and a facial expression module. Participants' responses were collected using a Likert-like scale to estimate the time interval between their head movement and the robot's response.

Datasets

Not discussed in the paper. The study uses a humanoid robot (iCub) as the interaction partner and its generated responses (facial expressions, gaze) as stimuli.

Key Insights and Ideas

  • Gaze is King (for SoA): The most striking finding is that the robot's gaze direction (mutual vs. averted) had a significant impact on participants' sense of agency, whereas the robot's facial expression (happy vs. sad) did not. This suggests that for implicit SoA in human-robot interaction, how the robot "looks" at you is more influential than how it "feels."
  • Postdictive Mechanisms at Play: The study hints that gaze influences SoA through postdictive mechanisms – how we retrospectively interpret the outcome of our actions. This is supported by the finding that when gaze was randomized (unpredictable), it affected SoA, suggesting it's not about predicting the outcome but rather about how we make sense of it after it happens.
  • The "Go" Signal Matters: In Experiment 3, where a neutral auditory cue ("Go") followed the social signals, the averted gaze led to a stronger SoA when it was randomized. This suggests that the combination of a neutral cue with a potentially salient social signal (averted gaze) can strongly influence our sense of control.
  • Contrasting Gaze Effects: Interestingly, the direction of the gaze effect on SoA was not consistent across all experiments. Experiment 2 showed stronger SoA with mutual gaze, while Experiment 3 (under specific conditions) showed stronger SoA with averted gaze. This suggests the effect of gaze might be more about signal strength or context than a simple positive/negative valence.

Key Results

  • Experiment 1 (Facial Expression): No significant difference in temporal estimates (and thus SoA) was found between the happy and sad facial expression conditions.
  • Experiment 2 (Gaze): Participants reported significantly lower temporal estimates (indicating higher SoA) when the robot engaged in mutual gaze compared to averted gaze.
  • Experiment 3 (Combined Effects):
    • There was no significant interaction between facial expression and gaze.
  • A main effect of gaze was found: averted gaze led to lower temporal estimates (higher SoA) than mutual gaze, but only when gaze was randomized.
  • When gaze was blocked (predictable), no significant difference was observed.
  • When facial expression was blocked (predictable), no significant difference was observed for gaze.
  • Separate analyses revealed that when the robot displayed a sad face and gaze was randomized, averted gaze led to lower temporal estimates (higher SoA) than mutual gaze.

Limitations and Future Work

  • Robot Expressiveness: The authors acknowledge that the iCub robot might not be sufficiently expressive to elicit strong effects from facial expressions. Future work could explore this with more expressive robots.
  • Predictive vs. Postdictive Mechanisms: While the study provides clues, it was not explicitly designed to definitively disentangle predictive and postdictive mechanisms. Future research could employ designs specifically tailored to isolate these processes.
  • Ecological Validity: While the study used natural actions (head turning), the temporal interval estimation task itself, even with a robot, might still differ from real-world social interactions.
  • Inconsistent Gaze Effects: The differing results on gaze direction between Experiment 2 and Experiment 3 highlight the complexity of social signals. Future work should investigate the role of signal strength, context, and the interplay with other cues (like auditory signals) in modulating SoA.
  • Methodological Differences: The authors suggest that the temporal interval estimation paradigm might be less sensitive than the Intentional Binding (IB) paradigm (Libet clock method) for capturing subtle effects of emotional expressions. Future studies could use the IB paradigm for comparison.

Summary updated

Abstract

Sense of Agency (SoA) is the feeling of control over one's actions and their outcomes. A well-established implicit measure of SoA is the temporal interval estimation paradigm, in which participants estimate the time interval between a voluntary action and its sensory consequence. In the present study, we aimed to investigate whether the valence of action outcome modulated implicit SoA. The valence was manipulated through interaction partner's (i) positive/negative facial expression, or (ii) type of gaze (gaze contact or averted gaze). The interaction partner was the humanoid robot iCub. In Experiment 1, participants estimated the time interval between the onset of their action (head movement towards the robot), and the robot's facial expression (happy vs. sad face). Experiment 2 was identical, but the outcome of participants' action was the type of robot's gaze (gaze contact vs. averted). In Experiment 3, we assessed-in a within-subject design-the combined effect of robot's type of facial expression and type of gaze. Results showed that, while the robot's facial expression did not affect participants' SoA (Experiment 1), the type of gaze affected SoA in both Experiment 2 and Experiment 3. Overall, our findings showed that the robot's gaze is a more potent factor than facial expression in modulating participants' implicit SoA.
BibTeX
@inproceedings{lombardi2023impact,
  author = {Maria Lombardi and Cecilia Roselli and Kyveli Kompatsiari and Federico Rospo and Lorenzo Natale and Agnieszka Wykowska},
  title = {{The impact of facial expression and communicative gaze of a humanoid robot on individual Sense of Agency}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-36864-0},
}
Figure