The Construction of Autonomous Virtual Hand in Virtual Assembly Environments

Ting Zhang, Cheng Cheng, Wei Ge, Jing ZhangInternational Journal of Computer Theory and Engineering 2017 · DOI

Abstract

This paper introduces a method of constructing autonomous virtual hand in virtual assembly environments. The goal of this research is presented to improve the practicability of virtual hand in virtual environments. The autonomous virtual hand is realized as the agent by three steps. A gesture set of virtual hand is constructed. Initial gesture, basic gesture and grasping gestures help virtual hand to achieve autonomous grasping. Perceptional ring is constructed for virtual hand, which support the autonomous virtual hand sense the virtual object. The system divides autonomous grasping process into three perception stages. By using this method, user heavy load will be greatly reduced in the assembly process. We prove that this autonomous grasping behavior of flexible virtual hand has a good availability and efficiency.

Introduction

In virtual reality (VR) and engineering assembly simulations, the "virtual hand" serves as the primary interface for interaction. Traditionally, these interactions are mediated through data gloves or image-based hand tracking. However, these methods often suffer from high latency, significant user fatigue due to the manual control of every finger joint, and a lack of inherent intelligence. Traditional virtual hands typically rely on computationally expensive collision detection, which can lead to unrealistic jittering or object clipping.

To address these challenges, this research proposes transforming the virtual hand from a passive puppet into an autonomous agent. This agent is designed to understand its environment and autonomously execute grasping maneuvers based on the geometry of target objects, thereby reducing the cognitive and physical load on the user.

Methods: The VirtualHand3D Framework

The researchers developed the VirtualHand3D system, which operates on an intent-based interaction model. Instead of direct control, the user provides high-level intent via a 3D mouse, while the system manages the micro-movements required for precise grasping.

The VirtualHand3D system loop, showing the interaction between user intent and autonomous perception stages.

The construction of the autonomous virtual hand involves three primary components:

1. The Gesture Set

A library of eight basic gestures was defined to match specific geometric features of assembly parts. For instance, the 2fNip gesture is used for small parts, AGrab for cylinders, and Grasp for spheres. These gestures are mapped to features such as convexity, concavity, shape, and size.

The library of eight basic gestures used for autonomous grasping of various geometric features.

2. The Perceptional Ring

To facilitate precise contact, each finger segment is equipped with a "perceptional ring." This virtual sensor acts as a distance detector between the finger and the object's surface. This allows the hand to stop movement exactly when contact (Face Mating) is achieved, ensuring a stable and realistic grasp without the jitter common in standard collision detection.

3. Three-Stage Perception Pipeline

The autonomous grasping process is divided into three sequential stages:

  • Feature Matching: The system identifies the target part's geometric features and selects the corresponding gesture from the library.
  • Coincidence Perception: The virtual hand automatically adjusts its pose and position to align with the target object.
  • Face Mating Perception: The fingers close autonomously until the perceptional rings signal that the grasp is complete.

Evaluation and Results

The system was tested using a custom environment within VirtualHand3D, featuring 145 assembly parts with 15 distinct geometric types (e.g., gears, shafts, prisms). The autonomous agent was compared against a traditional data glove-based system to establish performance benchmarks.

Key results include:

  • Performance: The autonomous hand achieved a real-time frame rate of 70 FPS, significantly outperforming the traditional collision-detection method, which dropped to 1 FPS under similar complexity.
  • Operational Load: User effort was reduced by over 75%. The system replaces the need for 14 manual sensor inputs from a glove with a single autonomous command.
  • Reliability and Semantics: The system achieved "Perfect" reliability ratings. Furthermore, it provided richer semantic feedback because the agent "understands" the object type it is interacting with, rather than merely detecting a point of contact.

Conclusion

The VirtualHand3D system demonstrates that shifting from direct control to intent-based interaction can significantly enhance the efficiency and usability of virtual assembly environments. While currently limited to a set of eight gestures and a 3D mouse interface, future work will focus on expanding the gesture library for more complex organic shapes and integrating predictive "intent-driven" algorithms to further streamline the user experience.

Summary updated

Abstract

This paper introduces a method of constructing autonomous virtual hand in virtual assembly environments. The goal of this research is presented to improve the practicability of virtual hand in virtual environments. The autonomous virtual hand is realized as the agent by three steps. A gesture set of virtual hand is constructed. Initial gesture, basic gesture and grasping gestures help virtual hand to achieve autonomous grasping. Perceptional ring is constructed for virtual hand, which support the autonomous virtual hand sense the virtual object. The system divides autonomous grasping process into three perception stages. By using this method, user heavy load will be greatly reduced in the assembly process. We prove that this autonomous grasping behavior of flexible virtual hand has a good availability and efficiency.
BibTeX
@article{zhang2017construction,
  author = {Ting Zhang and Cheng Cheng and Wei Ge and Jing Zhang},
  title = {{The Construction of Autonomous Virtual Hand in Virtual Assembly Environments}},
  journal = {International Journal of Computer Theory and Engineering},
  year = {2017},
  doi = {10.7763/ijcte.2017.v9.1137},
}
Figure