Abstract
Creating virtual humans with embodied, human-like perceptual and actuation constraints has the promise to provide an integrated simulation platform for many scientific and engineering applications. We present Dynamic and Autonomous Simulated Human (DASH), an embodied virtual human that, given natural language commands, performs grasp-and-stack tasks in a physically-simulated cluttered environment solely using its own visual perception, proprioception, and touch, without requiring human motion data. By factoring the DASH system into a vision module, a language module, and manipulation modules of two skill categories, we can mix and match analytical and machine learning techniques for different modules so that DASH is able to not only perform randomly arranged tasks with a high success rate, but also do so under anthropomorphic constraints and with fluid and diverse motions. The modular design also favors analysis and extensibility to more complex manipulation skills.
1. Introduction
In the quest for Embodied AI, researchers often face a fundamental trade-off: systems either possess high-level cognitive "brains" (logic and language) but simplified "bodies" (teleporting or using basic grippers), or they feature realistic physics but lack the ability to perceive and reason about their surroundings. DASH (Dynamic and Autonomous Simulated Human) bridges this gap by creating an autonomous agent—modeled after a five-year-old boy—that operates under human-like constraints.
Unlike many existing virtual human simulations that rely on pre-recorded motion capture (MoCap) data, DASH adapts to new, cluttered environments using only its own sensors: egocentric vision, proprioception, and touch.
2. The Challenge of Embodied AI
Most robotic systems utilize "ground-truth" data—perfect knowledge of object coordinates—rather than the noisy, partial information provided by a human-like eye. Furthermore, MoCap-based systems are often limited in their ability to adapt to randomized scenes. DASH addresses these issues by controlling a complex 29-degree-of-freedom (DoF) arm and hand using a modularized architecture that processes raw RGB egocentric camera feeds and verbal instructions.
3. Modular Architecture
The core innovation of DASH is its modularized design, which allows for the integration of specialized tools for distinct sub-tasks:
- Vision Module: Employs a Mask R-CNN for object segmentation and a ResNet-18 backbone to estimate object attributes (shape, color, 3D position) from the egocentric feed.
- Language Module: Uses a neural dependency parser to translate natural language (e.g., "put the blue cylinder on the red box") into logical task parameters.
- Manipulation Modules: Movement is bifurcated into two categories:
- Transition Stages (Reaching/Transporting): Solved using analytical motion planning (BiRRT) to ensure collision-free paths through cluttered spaces.
- Interaction Stages (Grasping/Stacking): Solved using Deep Reinforcement Learning (PPO) to handle contact-rich interactions.
4. Manipulation: Transition vs. Interaction
By separating movement into transition and interaction stages, the system avoids the "combinatorial explosion" associated with attempting to train a single neural network for end-to-end control.
This hybrid approach allows DASH to develop fluid, diverse strategies for handling objects of different shapes while maintaining the safety and efficiency of analytical planning during high-speed movements.
5. Key Results and Generalization
DASH achieved a 76% success rate using its noisy vision system, which is remarkably close to the 79% success rate achieved with ground-truth data. When excluding trials where a collision-free path was mathematically impossible (motion planning failures), the success rate rose to 85.8%.
Beyond performance metrics, DASH demonstrates significant zero-shot generalization. The DRL grasping policy, though trained on primitive shapes, successfully handled complex, unseen geometries like the Stanford Bunny. Additionally, the closed-loop policy allows for emergent "rescue" behaviors, such as adjusting a grip if an object slips or stabilizing a falling stack.
6. Conclusion and Future Work
DASH represents a significant step toward realistic human simulation. However, limitations remain, including a fixed torso that restricts workspace and a vision system that currently struggles with non-primitive shapes. Future work will explore whole-body motion, dual-arm coordination, and more sophisticated language modules capable of handling ambiguity through dialogue.
Summary updated