Reinforcement learning with digital human models of varying visual characteristics

Nitesh Bhatia, Ciara Pike-Burke, Eduardo Normando, Omar K. Matar2022 · DOI

TL;DR

This paper introduces a novel approach to simulate human visual behavior in virtual environments by integrating Digital Human Models (DHMs) with reinforcement learning (RL). By equipping DHMs with different "vision personas" (varying acuity, field of view, etc.), the study demonstrates how visual impairments can impact task performance in simulated target search and wayfinding scenarios, offering a more realistic and nuanced understanding of human-computer interaction.

What Problem Does This Paper Address?

Traditional Digital Human Modeling (DHM) often focuses on physical ergonomics (e.g., reach, fit, posture) but falls short in accurately simulating the impact of visual capabilities on task performance. This is a significant gap, as vision is crucial for tasks like target search and wayfinding. Existing DHM tools offer limited qualitative visualization of vision, failing to capture the nuances of how varying visual acuity, field of view, and focus affect an individual's ability to perceive and interact with their environment. This limitation hinders the design of truly vision-friendly products and workspaces, especially considering the prevalence of visual impairments in the population. The paper aims to bridge this gap by developing a DHM framework that can simulate diverse visual characteristics and their influence on task performance.

Methods

The core of this research lies in creating a reinforcement learning agent based on a Digital Human Model (DHM) within the Unity 3D game engine. The framework comprises three key modules:

  1. Low-level Motion Synthesis: Controls the DHM's movement (forward, backward, left, right) and animation.
  2. High-level Decision-Making: Utilizes a deep reinforcement learning policy (specifically Proximal Policy Optimization - PPO) to guide the DHM's actions based on visual input, aiming to move towards a target.
  3. Functional Vision Module: This is the novel component. It simulates human vision using a set of four cameras positioned at the DHM's eye level. By adjusting camera properties like Field of View (FoV), resolution, and focus (near/far clip planes), the researchers created three distinct "vision personas":
  4. 'Good Vision': Mimics natural human vision with high acuity in central FoV and lower acuity in peripheral FoV.
  5. 'Poor Vision' Type 1: Simulates low acuity across all FoV regions.
  6. 'Poor Vision' Type 2: Models high myopia, with reduced focus range and varying acuity.

The DHM is trained using Unity's ML-Agents package, which interfaces with TensorFlow for the RL training. The task is framed as a Markov Decision Process (MDP) where the DHM receives rewards for reaching the target and penalties for hitting distractors or exceeding a time limit.

Datasets

The paper does not explicitly mention the use of external datasets for training or evaluation. Instead, it describes the creation of a custom virtual workspace within the Unity game engine. This workspace contains a target (a Landolt C) and a distractor (a mirror-inverted Landolt C) that appear interchangeably on walls. The DHM's movement and interactions within this self-contained virtual environment constitute the training and evaluation "dataset."

Key Insights and Ideas

  • Vision as a Modifiable Agent Trait: The most significant insight is the successful integration of variable visual characteristics directly into an RL agent's "perception" system. This allows for a more nuanced simulation of human behavior than traditional DHM, which often treats vision as a static property.
  • Impact of Visual Impairments on Task Performance: The study clearly demonstrates that even subtle variations in vision (e.g., reduced acuity or focus) can lead to significant differences in how an agent navigates, searches for targets, and completes tasks. This highlights the importance of considering visual health in design.
  • RL as a Tool for Simulating Complex Human Behavior: The research showcases the power of deep reinforcement learning to train agents to perform complex tasks like wayfinding, learning emergent behaviors that are influenced by their simulated sensory inputs.
  • "Decision-Making Zones" Emerge from Heatmaps: The heatmap visualizations of DHM movement reveal distinct "hot spots" that correspond to decision-making areas, providing qualitative insights into the agent's strategy based on its vision persona.

Key Results

  • Behavioral Differences: The three vision personas exhibited distinct movement patterns and exploration strategies. The 'good vision' persona was most efficient, while 'poor vision' type 2 showed more exploratory behavior, moving forward until a target was visible.
  • Action Efficiency: The 'good vision' persona required the fewest actions per trial (around 993 correct actions out of 1000 trials), while 'poor vision' type 2 required nearly twice as many actions.
  • Task Duration: The 'good vision' persona completed tasks in an average of 5.1 seconds, significantly faster than 'poor vision' type 1 (10.03 seconds) and 'poor vision' type 2 (12.68 seconds).
  • Training Time: Training the 'good vision' persona was fastest (approx. 1.2 hours), while 'poor vision' personas required up to 6 hours of training on a consumer-grade GPU. This demonstrates that RL models for vision-impaired agents can be trained within a reasonable timeframe.

Limitations and Future Work

  • Simplified Vision Model: The paper acknowledges that the functional vision model is a simplification of the complex human visual system. While it captures key aspects like acuity and FoV, it doesn't fully replicate all nuances of human perception and cognition.
  • Task Simplicity: The target search and wayfinding task, while effective for demonstrating the concept, is relatively simple. More complex tasks would be needed to fully explore the capabilities and limitations of the framework.
  • No Human Data for Direct Comparison: The study relies on simulated DHM behavior rather than direct comparison with human subjects performing the same tasks.
  • Future Work: The authors plan to enhance the framework by:
  • Integrating human-in-the-loop deep reinforcement learning to bring the DHM behavior closer to natural human behavior.
  • Incorporating data from virtual reality-based user studies.
  • Exploring integrations of trained behavior policies with existing DHM tools like Jack.

Summary updated

Abstract

Digital Human Modelling (DHM) is rapidly emerging as one of the most cost-effective tools for generating computer-based virtual human-in-the-loop simulations. These help better understand individual and crowd behaviour under complex situations. For tasks such as target search and wayfinding, the eye is the primary channel for processing perceptual information and decision making. Existing experimental human studies in the literature have highlighted the relationship between the field of vision, visual acuity, accommodation, and its effect on visual search performance. This paper presents a methodology for the simulation of visual behaviour in target search and a wayfinding task by employing DHM as a reinforcement learning agent with functional vision characteristics. We used Unity 3D game engine to build the DHM and virtual workspace, Unity ML-Agents package to realise its connection with TensorFlow, and the Proximal Policy Optimization (PPO) algorithm to train DHM in finding a target through intensive reinforcement learning (RL). For the functional vision system, we have considered three human-inspired vision personas: (i) ‘good vision’, (ii) ‘poor vision’ type 1 (low acuity like), and (iii) ‘poor vision’ type 2 (high myopia like). We have compared the emergent behaviour of DHM for each of the three personas and RL training performance. The results conclude that simulating reinforcement learning agents with varying vision characteristics can evaluate their impact on visual task performance.
BibTeX
@misc{bhatia2022reinforcement,
  author = {Nitesh Bhatia and Ciara Pike-Burke and Eduardo Normando and Omar K. Matar},
  title = {{Reinforcement learning with digital human models of varying visual characteristics}},
  year = {2022},
  doi = {10.17077/dhm.31782},
}
Figure