Towards estimating affective states in Virtual Reality based on behavioral data

Valentin Holzwarth, Johannes Schneider, Joshua Handali, Joy Gisler, Christian Hirt, Andreas M. Kunz, Jan vom Brocke

Abstract

This research investigates the potential for estimating a user’s affective and cognitive states in Virtual Reality (VR) through non-intrusive behavioral tracking. By focusing on head yaw (side-to-side rotation) during a seated assembly task, the study identifies significant correlations between movement patterns and subjective experiences such as mental demand, presence, and perceived performance. The findings suggest that real-time behavioral data can not only predict user states with reasonable accuracy but also serve as a tool for validating the truthfulness of post-experience surveys.

Introduction

In the field of Virtual Reality, understanding the user's internal state—such as their cognitive workload or sense of presence—is critical for designing effective experiences. Traditionally, researchers rely on post-session questionnaires like the NASA-TLX or the System Usability Scale (SUS). However, these "gold standard" methods are inherently flawed: they are asynchronous, relying on memory; they are subjective and prone to bias; and they are intrusive, often requiring the interruption of the immersive experience.

This paper proposes a shift toward synchronous, non-intrusive behavioral measures. By leveraging data already collected by VR hardware, such as head tracking, developers can build adaptive systems that respond to a user’s needs in real-time without breaking immersion.

Methods

The researchers conducted a user study with 42 participants (26 male, 16 female) who performed a manual assembly task within a Virtual Training Environment (VTE). The task required participants to interpret 2D technical plans and assemble 3D components while seated.

The Virtual Training Environment (VTE) used in the study, showing the 2D plans and assembly area.

Behavioral Tracking and Analysis

The study focused specifically on head yaw (rotation around the vertical axis), a metric available on nearly all VR hardware. Three primary features were calculated from the 2 Hz sampled data:

  • Mean angular displacement ($Me$): The average position of the head.
  • Standard deviation of displacement ($Cv$): A measure of the range of movement.
  • Mean angular speed ($Vh$): How quickly the user rotated their head.

Following the VR session, participants completed standardized questionnaires to establish "ground truth" for Presence, NASA-TLX (mental/physical demand, effort, frustration), and Usability.

Results and Key Insights

Linear regression analysis revealed that head movement features could explain up to 25% of the variance in user-reported metrics. The study identified several distinct behavioral patterns:

  • The "Search" Pattern: Users experiencing high mental demand exhibited a higher standard deviation in head displacement ($Cv$) but lower angular speed ($Vh$). This suggests a pattern of extensive but slow searching for information within the environment.
  • Confidence and Speed: High perceived performance was strongly correlated with faster head movements ($Vh$). Users who felt confident in their task moved between points of interest with more "snappy" and certain rotations.
  • Presence and Orientation: A correlation was found between head tilt and presence. In this specific VTE, looking toward the instructions (on the right) rather than the task (on the left) was associated with lower reported presence, as it drew the user's attention toward meta-information.
Regression analysis results showing the coefficients and significance levels for various user state measures.

Predictive Modeling and Dishonesty Detection

The team trained Decision Tree and Linear Regression models to classify user states. The Decision Tree achieved 66.7% accuracy for physical demand and 64.3% for mental demand. Furthermore, the models were used to detect "manipulated" or fake questionnaire answers. By comparing reported scores against predicted behavioral patterns, the system could identify dishonest responses with 60% to 85% accuracy, with higher accuracy for more significant discrepancies.

Discussion and Conclusion

While the study was limited to a seated environment and a single axis of head movement, the results provide a compelling proof-of-concept for behavioral affective estimation. The ability to detect user states and validate survey data using low-cost, hardware-agnostic metrics like head yaw opens new doors for adaptive VR training and gaming.

Future work will look to expand the feature set to include eye-tracking and hand-tracking, as well as exploring movement patterns in room-scale VR environments where users are free to walk and stand. Additionally, larger datasets will be required to apply more complex deep learning techniques to further refine these predictive models.

Summary updated

Abstract

Inferring users’ perceptions of Virtual Environments (VEs) is essential for Virtual Reality (VR) research. Traditionally, this is achieved through assessing users’ affective states before and after being exposed to a VE, based on standardized, self-assessment questionnaires. The main disadvantage of questionnaires is their sequential administration, i.e., a user’s affective state is measured asynchronously to its generation within the VE. A synchronous measurement of users’ affective states would be highly favorable, e.g., in the context of adaptive systems. Drawing from nonverbal behavior research, we argue that behavioral measures could be a powerful approach to assess users’ affective states in VR. In this paper, we contribute by providing methods and measures evaluated in a user study involving 42 participants to assess a users’ affective states by measuring head movements during VR exposure. We show that head yaw significantly correlates with presence, mental and physical demand, perceived performance, and system usability. We also exploit the identified relationships for two practical tasks that are based on head yaw: (1) predicting a user’s affective state, and (2) detecting manipulated questionnaire answers, i.e., answers that are possibly non-truthful. We found that affective states can be predicted significantly better than a naive estimate for mental demand, physical demand, perceived performance, and usability. Further, manipulated or non-truthful answers can also be estimated significantly better than by a naive approach. These findings mark an initial step in the development of novel methods to assess user perception of VEs.
BibTeX
@article{holzwarth2021towards,
  author = {Valentin Holzwarth and Johannes Schneider and Joshua Handali and Joy Gisler and Christian Hirt and Andreas M. Kunz and Jan vom Brocke},
  title = {{Towards estimating affective states in Virtual Reality based on behavioral data}},
  journal = {Virtual Real.},
  year = {2021},
  doi = {10.1007/s10055-021-00518-1},
}
Figure