Data-driven camera manipulation about vertical locomotion in a virtual environment

Seungwon Seo, Seong-Rae Noh, RoUn Lee, Seung-Jun Park, HyeongYeop KangJournal of the Korea Computer Graphics Society 2022 · DOI

Abstract

In this paper, the goal is to investigate how manipulating the camera can minimize motion sickness and maximize immersion when a user moves in a virtual environment that requires vertical movement. In general, since a user uses virtual reality in a flat space, the actual movement of the user and the virtual movement are different, resulting in sensory conflict, which has the possibility of causing virtual reality motion sickness. Therefore, we propose three powerful camera manipulation techniques, implement them, and then propose which model is most appropriate through user experiments.

Introduction

The core challenge in Virtual Reality (VR) locomotion is Sensory Conflict Theory. When a user navigates a virtual environment that involves verticality—such as climbing stairs or ascending hills—while remaining on a flat physical floor, a mismatch occurs between visual stimuli and the vestibular system. The eyes perceive ascent, while the inner ear reports level ground, frequently resulting in "VR Sickness." While horizontal movement techniques like "Redirected Walking" are well-documented, vertical locomotion remains a significant barrier to immersion. This study explores whether data-driven machine learning models or heuristic algorithms can effectively manipulate the virtual camera to mitigate these effects.

Methodology

The researchers evaluated three distinct approaches to controlling virtual camera height during vertical movement:

1. Real-world Data Model (Test 1)

Motion capture data was collected from a participant using a VIVE HMD and trackers on the waist and feet while navigating a physical three-step staircase. This data was used to train a Multi-Layer Perceptron (MLP) to predict head and waist height based on foot positions over the preceding 10 frames.

Experimental setup for real-world data collection involving physical stairs.

2. Virtual Data Model (Test 2)

To overcome the limitations of physical data collection, a synthetic dataset was generated using Unity’s "Foot IK" (Inverse Kinematics). A virtual character was simulated walking on diverse terrains. To ensure robustness, the character's height and width were dynamically scaled using Gaussian random variables, creating a more generalized dataset than the real-world capture.

3. Heuristic Algorithm (Test 3)

A non-learning approach was implemented where the camera height was adjusted simply based on the height of the lower foot on the virtual terrain.

Model Architecture

The MLP models utilized an Exponential Linear Unit (ELU) activation function and were optimized using the Adam algorithm to minimize Mean Squared Error (MSE). The input layer consisted of 20 nodes representing foot positions, feeding into two hidden layers of 128 nodes each.

The Multi-Layer Perceptron (MLP) architecture used for height prediction.

Results and Discussion

The study evaluated seven participants using the Simulator Sickness Questionnaire (SSQ) and the Igroup Presence Questionnaire (IPQ).

The "Jitter" Trap: Surprisingly, the sophisticated MLP models (Tests 1 and 2) induced higher levels of motion sickness than the heuristic algorithm. The neural networks produced subtle vertical oscillations—micro-vibrations—as they predicted height frame-by-frame. In VR, these high-frequency camera shakes are highly provocative for nausea.

Synthetic vs. Real Data: Between the two machine learning approaches, the model trained on virtual data (Test 2) outperformed the real-world model (Test 1). The synthetic dataset's diversity, achieved through random scaling of body types, allowed for better generalization across different users.

Consistency and Immersion: The heuristic algorithm (Test 3) was rated as the most "natural" and immersive. By following the lower foot, the camera movement remained predictable and smooth, lacking the "micro-stutter" inherent in the AI-driven predictions.

Conclusion

While the real-world ML model achieved the highest mathematical accuracy (0.86cm error for head height), it failed to provide the best user experience due to temporal instability. The heuristic approach proved superior for current applications. Future research will focus on implementing temporal smoothing filters or utilizing recurrent architectures like LSTMs to eliminate jitter, as well as testing on more complex, irregular terrains.

Summary updated

Abstract

In this paper, the goal is to investigate how manipulating the camera can minimize motion sickness and maximize immersion when a user moves in a virtual environment that requires vertical movement. In general, since a user uses virtual reality in a flat space, the actual movement of the user and the virtual movement are different, resulting in sensory conflict, which has the possibility of causing virtual reality motion sickness. Therefore, we propose three powerful camera manipulation techniques, implement them, and then propose which model is most appropriate through user experiments.
BibTeX
@article{seo2022data,
  author = {Seungwon Seo and Seong-Rae Noh and RoUn Lee and Seung-Jun Park and HyeongYeop Kang},
  title = {{Data-driven camera manipulation about vertical locomotion in a virtual environment}},
  journal = {Journal of the Korea Computer Graphics Society},
  year = {2022},
  doi = {10.15701/kcgs.2022.28.3.13},
}
Figure