Affective evaluation of multimodal dialogue games for preschoolers using physiological signals

Vassiliki Kouloumenta, Manolis Perakakis, Alexandros PotamianosInterspeech 2013 · OpenReview

Abstract

This study investigates the use of consumer-grade, single-sensor EEG technology to monitor the cognitive and emotional states of preschoolers and adults during multimodal educational gameplay. By analyzing "attention" and "meditation" levels across speech- and mouse-based interfaces, the research demonstrates that low-cost physiological sensors can reliably track internal states that traditional performance metrics often miss. Findings reveal that children and adults exhibit remarkably similar cognitive patterns when navigating multimodal systems, suggesting early stabilization of the underlying cognitive architecture for such interactions.

Introduction

Designing multimodal dialogue systems—which integrate speech, touch, and graphics—for young children presents unique challenges. Traditional evaluation metrics, such as task completion time or error rates, provide a surface-level view of success but fail to capture a child's internal state, such as frustration or focus. Affective computing offers a potential solution, yet it is rarely applied to pediatric populations due to the technical difficulties of child-speech recognition and the physical discomfort of medical-grade EEG equipment. This paper explores whether non-invasive, low-cost physiological sensors can provide a reliable "window" into a child's cognitive state during play.

Methodology

To evaluate internal states, researchers developed a suite of five educational games tailored for preschoolers, covering tasks such as animal recognition, shape identification, and basic addition.

Figure 1: Examples of the educational game suite, including (a) addition, (b) shape recognition, and (c) farm animal identification.

Technical Implementation

The study utilized a Wizard of Oz (WoZ) setup, where a human operator simulated the speech recognition system to ensure 100% accuracy, thereby isolating affective responses from the frustration of technical failures.

Key Components: * Hardware: The NeuroSky MindSet, a single-electrode dry EEG sensor worn on the forehead, was selected for its child-friendly design. * Metrics: The system tracked "Attention" (mental focus) and "Meditation" (mental calmness), alongside a calculated "Arousal" metric based on the Beta-to-Alpha brainwave ratio ($\beta/\alpha$). * Segmentation: Interaction turns were divided into four phases: Waiting for Prompt (WP), System Prompt (P), Waiting for Answer (WA), and Active Answer (A).

Experimental Design

The pilot study involved two distinct populations to compare physiological patterns:

  • Children: 6 preschoolers (ages 5.5 to 6.5).
  • Adults: 14 individuals (ages 20 to 59) serving as a control group.

In total, 80 interaction sessions were recorded, providing a robust dataset for comparing cognitive loads across different age groups and interaction modalities.

Results and Discussion

Modality Symmetry and Attention

A primary finding was the trade-off between visual and auditory attention. When using a mouse, attention peaked during the "Active" (A) phase. Conversely, when using speech, attention peaked during the "Prompt" (P) phase as users listened to instructions. This suggests the brain dynamically allocates focus to the primary input channel required by the specific modality.

Figure 2: Comparison of physiological signals across interaction phases (WP, P, WA, A) for mouse-based and voice-based modalities.

Cognitive Load and Task Type

Adults consistently showed higher "Meditation" values than children, indicating that the preschool-level tasks imposed a significantly lower cognitive load on the control group. Furthermore, visual recognition tasks (e.g., identifying farm animals) elicited higher attention levels than computational tasks (e.g., addition). This is likely due to the placement of the forehead electrode over the frontal cortex, which is heavily involved in visual processing.

Performance Correlations

The study found a significant negative correlation (-0.27) between activity time and arousal; higher arousal levels were associated with faster response times. Additionally, attention levels during the "Waiting for Answer" (WA) phase were positively correlated (0.15) with correct answers, suggesting that increased mental effort during the thinking phase directly impacts performance.

Conclusion

This research proves that single-sensor EEG is a feasible tool for capturing consistent variations in a child's internal state during multimodal interactions. Despite the age difference, the patterns of brain activity across interaction turns were nearly identical between children and adults, suggesting that the fundamental cognitive architecture for multimodal interaction is established early in development.

Future work will focus on integrating these physiological signals into "closed-loop" systems, where game difficulty or character behavior adapts in real-time to the user's detected attention or frustration levels. Additionally, moving beyond the WoZ setup to study the impact of actual speech recognition errors will provide deeper insights into how children cope with technological misunderstandings.

Summary updated

Abstract

In this pilot study, we investigate the differences in the electroencephalography (EEG) signal patterns of children and adults while interacting with a multimodal dialogue computer game. The gaming application is designed for preschoolers, implements five popular learning tasks and has variable levels of difficulty. In this pilot, to simplify the data collection process for young children, we use the NeuroSky MindSet device which is a single forehead dry sensor device. The raw signals and the estimated attention, meditation and arousal signals are analyzed during the interaction and compared for adult and children user populations. Results show consistent variations as a function of modality used (speech vs mouse input), difficulty level and task success. The physiological signal pattern within an interaction turn is also estimated and analyzed. Overall, children and adults demonstrated very similar physiological signal patterns during multimodal interaction.
BibTeX
@inproceedings{kouloumenta2013affective,
  author = {Vassiliki Kouloumenta and Manolis Perakakis and Alexandros Potamianos},
  title = {{Affective evaluation of multimodal dialogue games for preschoolers using physiological signals}},
  booktitle = {Interspeech},
  year = {2013},
  url = {https://openreview.net/forum?id=6LsjB2A8ta},
}
Figure