An Embodied Tutoring System for Literal vs. Metaphorical Concepts

Marietta Sionti, Thomas Schack, Yiannis AloimonosFrontiers in Psychology 2018 · DOI

Abstract

In this paper, we combine motion-captured data with linguistic notions in a game-like tutoring system to help elementary school students better differentiate literal from metaphorical uses of motion verbs based on embodied information. In addition to the thematic goal, we intend to improve young students' attention and spatiotemporal memory by presenting sensorimotor data experimentally collected from thirty-two participants. Furthermore, we examine the accomplishment of the tutor's goals and compare them to the standard curriculum approach. Sixty-nine elementary school students were randomly divided into two experimental groups (game-like and traditional) and one control group. All groups were tested in pre- and post-tests. Two-way analysis of variance suggests that the experimental groups showed progress, with the game-like group showing fewer wrong answers and higher learning achievements. Furthermore, in the game-like condition, participants needed gradually shorter periods of time to identify the avatar's actions, indicating improved attentional and spatiotemporal memory.

Introduction: Bridging the Concrete and the Abstract

For elementary school students, transitioning from a concrete, physical understanding of the world to the nuances of figurative language represents a significant developmental milestone. Distinguishing between a literal action, such as "lifting a box," and its metaphorical counterpart, "lifting a family's spirits," requires a cognitive leap that traditional text-based education often struggles to facilitate.

The theory of embodied cognition posits that our brains process abstract concepts by mentally simulating the physical actions they are derived from. Despite this, there is a notable gap in educational tools that leverage actual sensorimotor data to bridge this divide. This research addresses two primary challenges: helping students differentiate literal from metaphorical meaning through physical grounding and improving underlying cognitive faculties like attention and spatiotemporal memory through the observation of biological motion.

Methodology: Grounding Language in Motion

The researchers developed an Intelligent Tutoring System (ITS) titled "Playing with the Stars." The system's design is rooted in the use of high-fidelity 3D motion-capture (MoCap) data to provide a "ground truth" for physical verbs.

Motion Capture and Point-Light Displays

The team recorded 32 native speakers of English, Greek, and German performing specific actions (e.g., lift, turn, trigger) using Xsens MVN suits and Vicon camera systems. Unlike standard MoCap databases that record continuous sequences, this dataset focused on the "conceptual representation" of individual verbs to ensure the motion accurately reflected the linguistic core of the word across different languages.

To focus the students' attention on the kinematics of motion rather than visual distractions, the system utilizes point-light displays. These are moving dots representing major human joints.

Figure 1: The process of converting human motion into point-light displays for the tutoring system.

Scaffolding and Feedback

The ITS employs a "zone of proximal development" approach. If a student fails to identify an action from the point-light display, the system provides a hint and adds "bones"—lines connecting the dots—to make the human shape more recognizable. This feedback loop helps students realize how they are perceiving motion, fostering metacognitive awareness.

Experimental Design

A study was conducted with 69 sixth-grade students, randomly divided into three groups:

  1. Game-like Group (GG): Used the "Playing with the Stars" software.
  2. Traditional Group (TG): Used standard curriculum materials and verbal explanations.
  3. Control Group (CG): Received no intervention.

Results: Cognitive and Linguistic Gains

The effectiveness of the intervention was evaluated using a two-way ANOVA on pre- and post-test scores.

Learning Achievements

The Game-like Group (GG) demonstrated significantly higher learning achievements and fewer errors in linguistic tasks compared to both the Traditional and Control groups. By seeing a literal physical representation of a verb, students more easily recognized that metaphorical uses were non-physical extensions of that movement.

Figure 2: The 'Playing with the Stars' game interface showing a linguistic evaluation task.

Processing Speed and Memory

In the GG condition, the time required to identify actions dropped drastically over the course of the session. Initially, students took approximately 19 seconds to identify an action; by the final questions, this fell to between 5 and 8 seconds. This rapid improvement suggests a significant enhancement in spatiotemporal recognition and attentional focus.

Discussion and Future Work

While the results are promising, the authors acknowledge the pilot scale of the study. Future work aims to integrate machine learning modules to allow the system to adapt dynamically to an individual student’s cognitive pace. Additionally, the researchers suggest the system could be adapted for foreign language learning, where physical grounding could accelerate vocabulary acquisition. Further validation using EEG to measure the "neural signature" of these cognitive improvements is also planned.

Summary updated

Abstract

In this paper we combine motion captured data with linguistic notions (preliminary study) in a game-like tutoring system (study 1), in order to help elementary school students to better differentiate literal from metaphorical uses of motion verbs, based on embodied information. In addition to the thematic goal, we intend to improve young students' attention and spatiotemporal memory, by presenting sensorimotor data experimentally collected from thirty two participants in our motion capturing labs. Furthermore, we examine the accomplishment of tutor's goals and compare them to curriculum's approach (study 2). Sixty nine elementary school students were randomly divided in two experimental groups (game-like and traditional) and one control group, which did not undergo an intervention. All groups were tested in pre and post-tests. Even though the diagnostic pretests present a uniform picture, two way analysis of variance suggests that the experimental groups showed progress in post-tests and, more specifically, game-like group showed less wrong answers in the linguistics task and higher learning achievements compared to the other two groups. Furthermore, in the game-like condition the participants needed gradually shorter period of time to identify the avatar's actions. This finding was considered as a first indication of attentional and spatiotemporal memory's improvement, while the tutor's assistance features cultivated students' metacognitive perception.
BibTeX
@inproceedings{sionti2018embodied,
  author = {Marietta Sionti and Thomas Schack and Yiannis Aloimonos},
  title = {{An Embodied Tutoring System for Literal vs. Metaphorical Concepts}},
  booktitle = {Frontiers in Psychology},
  year = {2018},
  doi = {10.3389/fpsyg.2018.02254},
}
Figure