GAgent: An Adaptive Rigid-Soft Gripping Agent with Vision Language Models for Complex Lighting Environments

Zhuowei Li, Miao Zhang, Xiaotian Lin, Meng Yin, Shuai Lu, Xueqian WangarXiv 2024 · DOI · arXiv

Introduction

Robotic grasping in "open-world" environments—such as those encountered by search-and-rescue drones or autonomous service robots—remains a significant challenge in robotics. Two primary hurdles persist: environmental complexity and physical adaptability. Traditional computer vision systems often fail under non-ideal lighting conditions, such as extreme overexposure or near-darkness. Furthermore, mechanical grippers typically face a trade-off: rigid designs risk crushing delicate objects, while soft grippers often lack the structural stiffness required to lift heavy or industrial loads.

To address these limitations, we present GAgent, a novel robotic system that integrates a bionic, variable-stiffness soft gripper with a Vision-Language Model (VLM) "brain." By leveraging the high-level reasoning of Large Multimodal Models (LMMs) like GPT-4, GAgent can autonomously identify object materials and adjust its physical rigidity to ensure successful grasping across a wide variety of items and lighting conditions.

System Architecture and Workflow

The GAgent framework is built upon a sophisticated software stack designed to bridge the gap between visual perception and physical execution. The system operates through three primary modules:

  1. Prompt Engineer Module: This component translates high-level task descriptions and operational rules into structured prompts that the VLM can process.
  2. VLM Core (GPT-4.0): Serving as the central decision-maker, the VLM performs material analysis and estimates optimal grasp points based on visual input.
  3. Workflow Module: This module manages the execution pipeline, utilizing a "Chain of Thought" (CoT) strategy. Crucially, it incorporates a low-light enhancement algorithm (Scrnet) to clarify environmental images before they are processed by the VLM, ensuring robust performance in dark environments.
Figure 1: The GAgent system architecture, illustrating the interaction between the VLM core, the tool pool, and the multi-step workflow for detection and grasping.

Bionic Hybrid Gripper Design

The physical component of GAgent is a specialized bionic hybrid gripper inspired by the anatomy of the human hand. The design utilizes a combination of materials to achieve variable stiffness:

  • Bones: Rectangular springs provide the internal structural framework.
  • Flesh: A silicone outer layer ensures a soft, conforming interface for delicate objects.
  • Tendons: Motor-driven cables allow the system to apply pretension. By pulling these cables, the gripper can dynamically increase its stiffness, transitioning from a soft state to a more rigid configuration.
Figure 2: Detailed hardware design of the bionic hybrid gripper, showing the digital servo, winding reel, and embedded spring mechanism.

Experimental Evaluation and Key Insights

GAgent was evaluated using an open-world benchmark consisting of 42 diverse items, including daily necessities and industrial tools such as eggs, seaweed, hammers, and power clips. Tests were conducted across three lighting scenarios: normal daylight, noon overexposure, and evening low light.

Cognition-Physique Feedback Loop

A key insight from our research is the effectiveness of the feedback loop between the VLM and the hardware. The VLM does not merely detect an object; it infers material properties. For instance, when identifying a fragile peach, the VLM commands the gripper to adopt a low stiffness level. In cases of initial failure—such as misidentifying a hard durian shell as a soft fruit—the system's reasoning loop allows for self-correction, re-evaluating the material and adjusting stiffness for a successful second attempt.

Dynamic Stability

Beyond load capacity, the variable-stiffness mechanism significantly improves the gripper's dynamic stability. Pre-stretching the internal springs dampens vibrations, which is critical for drone-mounted applications. Experimental data showed that GAgent stabilized 28% faster (1.8s vs. 2.5s) and reduced vibration amplitude from 25° to 19° compared to non-pretensioned states.

Results and Conclusion

GAgent successfully grasped all 42 items in the benchmark, demonstrating exceptional generalization without the need for task-specific training data. The integration of the brightness adjustment tool proved vital; without it, the system failed to focus on objects in extreme lighting, whereas the enhanced workflow maintained high success rates. The hybrid design allows the agent to lift loads up to 700g while remaining gentle enough to handle delicate items like potato chips.

Future work will focus on integrating GAgent onto Unmanned Aerial Vehicles (UAVs) for active flight testing in extreme outdoor conditions and optimizing smaller, local models to reduce the latency currently associated with high-level VLMs like GPT-4.

Summary updated

BibTeX
@misc{li2024gagent,
  author = {Zhuowei Li and Miao Zhang and Xiaotian Lin and Meng Yin and Shuai Lu and Xueqian Wang},
  title = {{GAgent: An Adaptive Rigid-Soft Gripping Agent with Vision Language Models for Complex Lighting Environments}},
  year = {2024},
  eprint = {2403.10850},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.10850},
}
Figure