Metaversal Learning Environments: Measuring, predicting and improving interpersonal effectiveness

Arjun Nagendran, Scott Compton, William Follette, Artem Golenchenko, Anna Compton, Jonathan GrizouarXiv 2022 · DOI · arXiv

Abstract

Interpersonal effectiveness is a critical component of professional success, yet traditional training methods are often difficult to scale and quantify. This paper introduces Metaversal Learning Environments (MLEs)—immersive virtual reality spaces where users interact with human-controlled, AI-animated avatars to practice soft skills. The research demonstrates that interpersonal performance in these environments is measurable, improvable through targeted practice, and predictable using machine learning models focused on non-linguistic audio features.

Introduction

In fields ranging from healthcare to leadership, the ability to navigate complex social interactions is paramount. However, "soft skills" training has historically relied on expensive, inconsistent live role-playing. The emergence of the "Metaverse" offers a new frontier for experiential learning, but until now, robust frameworks for quantifying human behavior within these digital realms have been lacking.

A Metaversal Learning Environment (MLE) featuring an AI-animated avatar in a professional office setting.

This study investigates whether a virtual environment can elicit natural human behavior, provide accurate performance metrics, and leverage data to predict conversational success. By creating a hybrid human-AI architecture, the researchers provide a scalable model for professional development.

Methodology

The authors developed a dual-component architecture to facilitate realistic social simulations:

  1. The Authoritative Component: An expert "inhabiter" controls the avatar’s intent and provides real-time performance ratings via the IMPACT score (measuring positive, neutral, or negative interaction moments).
  2. The Non-authoritative Component: The interface through which the learner interacts with the avatar.

To ensure realism while reducing the cognitive load on the expert inhabiter, the researchers implemented Audio-based AI Animation. This system automatically generates facial expressions and micro-gestures based on the inhabiter's voice, ensuring non-verbal cues are synchronized without manual input. For predictive modeling, the team extracted 48 facial emotions and various acoustic features (such as pitch and jitter) to train a Support Vector Machine (SVM) classifier.

Data Collection and Analysis

The study generated a proprietary dataset through two phases:

  • Pilot Study: 59 employees at Mursion refined the software interface.
  • Main Study: 51 participants completed four distinct conversational scenarios (screening, two training sessions, and a post-assessment).

This resulted in 204 synchronized audio/video interactions. The researchers mapped emotional data onto a 2D "circumplex" space (Active/Passive vs. Positive/Negative) to analyze the behavioral "footprints" of the participants.

Key Insights

The "Growth Zone"

The most significant improvements were observed in participants who initially struggled. Those rated as "unsuccessful" in early rounds saw their average IMPACT scores rise from 46.15 to 79.39 by the final scenario (p < .0219). This suggests MLEs are highly effective for targeted remediation.

Self-Awareness as a Predictor

The data revealed a strong correlation between self-awareness and success. "Good Estimators"—those whose self-assessments aligned with expert ratings—achieved a 76.6% success rate, compared to only 50.9% for "Over-Estimators."

The Predictive Power of Audio

One of the most significant findings was that non-linguistic audio features (tone, pitch, rhythm) were nearly as predictive of success as full audio-video data. By focusing on a subset of 17 top audio features, the researchers achieved an AUC of ~0.72 and 68% accuracy while reducing data processing time by 85%.

Results and Implications

The study confirms that individuals interact with avatars naturally, exhibiting the same behavioral traits they would in the physical world. The ability to predict success using only audio signals has massive implications for privacy and computational efficiency in real-world training applications. Furthermore, the significant improvement in scores across sessions validates the MLE as a legitimate proxy for real-world skill acquisition.

Limitations and Future Work

While promising, the study was limited by a relatively small and homogeneous sample size. Additionally, the researchers noted the difficulty in distinguishing between "learning effects" (actual skill improvement) and "practice effects" (becoming more comfortable with VR hardware).

Future research will explore the integration of real-time AI "nudges" to help learners adjust their behavior mid-conversation, potentially accelerating the path to interpersonal mastery.

Summary updated

Abstract

Experiential learning has been known to be an engaging and effective modality for personal and professional development. The Metaverse provides ample opportunities for the creation of environments in which such experiential learning can occur. In this work, we introduce a novel architecture that combines Artificial intelligence and Virtual Reality to create a highly immersive and efficient learning experience using avatars. The framework allows us to measure the interpersonal effectiveness of an individual interacting with the avatar. We first present a small pilot study and its results which were used to enhance the framework. We then present a larger study using the enhanced framework to measure, assess, and predict the interpersonal effectiveness of individuals interacting with an avatar. Results reveal that individuals with deficits in their interpersonal effectiveness show a significant improvement in performance after multiple interactions with an avatar. The results also reveal that individuals interact naturally with avatars within this framework, and exhibit similar behavioral traits as they would in the real world. We use this as a basis to analyze the underlying audio and video data streams of individuals during these interactions. Finally, we extract relevant features from these data and present a machine-learning based approach to predict interpersonal effectiveness during human-avatar conversation. We conclude by discussing the implications of these findings to build beneficial applications for the real world.
BibTeX
@misc{nagendran2022metaversal,
  author = {Arjun Nagendran and Scott Compton and William Follette and Artem Golenchenko and Anna Compton and Jonathan Grizou},
  title = {{Metaversal Learning Environments: Measuring, predicting and improving interpersonal effectiveness}},
  year = {2022},
  eprint = {2205.02875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.02875},
}
Figure