TL;DR
This paper investigates how acoustic cues like loudness and pitch in child-child and child-robot interactions during a speech-controlled game relate to engagement. They found that acoustic synchrony between children is linked to higher engagement, a phenomenon not observed with robots due to their non-adaptive behavior, suggesting a path for more engaging human-robot interactions.
What Problem Does This Paper Address?
Creating engaging and natural human-computer interactions, especially with robots, is a significant challenge. For children, engagement is crucial for learning, enjoyment, and social rapport. While automatic recognition of engagement is an active research area, understanding how to promote it, particularly through interaction dynamics, is less explored. This paper focuses on the acoustic dimension of interaction, specifically how the prosody (loudness and pitch) of participants' speech can be coordinated and how this coordination relates to engagement levels in both peer-to-peer and child-robot scenarios. The goal is to lay the groundwork for robots that can better foster engagement in children.
Methods
The researchers employed a linear mixed-effects (LME) model to analyze the relationship between acoustic features and engagement. This statistical approach is well-suited for nested data structures (like multiple interactions within pairs) and helps account for individual differences between participants and pairs.
They extracted two key acoustic features:
- Loudness: Measured as normalized speech intensity.
- Fundamental Frequency (F0): Representing pitch, measured in Hz.
These features were extracted from speech segments corresponding to keywords ("go" and "jump") within the game. The LME model was used to predict one participant's acoustic features based on the other participant's features and the annotated engagement score (ES). Crucially, the model allowed them to examine:
- The direct association between participants' acoustic features (e.g., how one child's loudness relates to another's).
- The association between a participant's acoustic features and their engagement level.
- Whether the relationship between participants' acoustic features is moderated by engagement (i.e., does acoustic synchrony increase when engagement is high?).
Datasets
The study utilized data from two types of interactions:
- Child-Child Interactions: 62 children (aged 5-10) played a speech-controlled game called "Mole Madness" in pairs. This resulted in approximately 391 seconds of gameplay per pair.
- Child-Robot Interactions: 61 of the same children played "Mole Madness" with a robot named Sammy. Sammy's vocal behavior consisted of pre-recorded utterances, and its actions were not dynamically adapted to the child's input. These interactions lasted approximately 330 seconds on average.
The engagement levels were annotated by three female coders who rated the "willingness" of children to continue the activity on a 10-second segment basis.
Key Insights and Ideas
- Acoustic Synchrony Matters for Child-Child Engagement: The study found a significant positive association between the acoustic patterns (loudness and pitch) of children interacting with each other. This synchrony was further moderated by engagement, meaning that children who were more engaged also exhibited higher acoustic synchrony. This suggests that children naturally attune to each other's vocal cues when they are deeply involved in an activity.
- Robots Lack Natural Engagement Dynamics: In contrast to child-child interactions, the acoustic synchrony between a child and the robot was not consistently significant. This is attributed to the robot's pre-programmed, non-adaptive behavior. The robot's random selection of utterances meant it didn't mirror or respond to the child's vocal patterns, thus failing to establish the same kind of acoustic coordination.
- Engagement is Multimodal, but Acoustics Play a Role: While engagement is a complex, multimodal phenomenon (involving visual, verbal, and non-verbal cues), this research highlights the specific contribution of acoustic coordination. The findings suggest that focusing on these vocal dynamics could be a key lever for enhancing engagement.
- "Similar Engagement" Matters: The researchers found that the positive association between acoustic synchrony and engagement was even stronger when considering only those interaction segments where both children had similar engagement scores. This implies that mutual engagement is a strong driver of acoustic coordination.
Key Results
- Child-Child Interaction:
- Significant positive association between children's loudness and F0 features (β₁₀).
- Significant positive association between a child's acoustic features and their engagement level (β₂₀).
- A significant positive effect of engagement on the association between children's acoustic features (β₁₁), indicating that higher engagement leads to stronger acoustic synchrony. This effect was more pronounced in "Similar ES Segments."
- Child-Robot Interaction:
- Significant association between the child's acoustic features and their engagement level (β₂₀).
- The association between the child's and robot's acoustic features (β₁₀) was not consistently significant.
- The effect of engagement on the child-robot acoustic score association (β₁₁) was not significant, reinforcing the idea that the robot's non-adaptive behavior prevents this dynamic.
Limitations and Future Work
- Robot Adaptability: The primary limitation identified is the robot's lack of adaptive behavior. The current robot's responses were pre-programmed and randomly selected, preventing it from mirroring or responding to the child's vocal patterns.
- Future Work: The authors propose future work focused on:
- Automatic Robot Adaptation: Developing systems where the robot's vocal characteristics (loudness, pitch) can be automatically synthesized to match or respond to the child's vocal patterns. This could involve building systems that measure children's vocal cues and adapt the robot's voice accordingly.
- Expanding to Visual Cues: Incorporating visual cues into the analysis and interaction design.
- Time-Based Models: Developing models that can predict engagement and quantify synchrony over time, rather than just in discrete segments.
- Enhancing Rapport: Using these findings to build engagement and social rapport in child-robot interactions, potentially making robots better social partners and tutors for children.
Summary updated