Introduction
Robotic grasping in "open-world" environments—such as those encountered by search-and-rescue drones or autonomous service robots—remains a significant challenge in robotics. Two primary hurdles persist: environmental complexity and physical adaptability. Traditional computer vision systems often fail under non-ideal lighting conditions, such as extreme overexposure or near-darkness. Furthermore, mechanical grippers typically face a trade-off: rigid designs risk crushing delicate objects, while soft grippers often lack the structural stiffness required to lift heavy or industrial loads.
To address these limitations, we present GAgent, a novel robotic system that integrates a bionic, variable-stiffness soft gripper with a Vision-Language Model (VLM) "brain." By leveraging the high-level reasoning of Large Multimodal Models (LMMs) like GPT-4, GAgent can autonomously identify object materials and adjust its physical rigidity to ensure successful grasping across a wide variety of items and lighting conditions.
System Architecture and Workflow
The GAgent framework is built upon a sophisticated software stack designed to bridge the gap between visual perception and physical execution. The system operates through three primary modules:
- Prompt Engineer Module: This component translates high-level task descriptions and operational rules into structured prompts that the VLM can process.
- VLM Core (GPT-4.0): Serving as the central decision-maker, the VLM performs material analysis and estimates optimal grasp points based on visual input.
- Workflow Module: This module manages the execution pipeline, utilizing a "Chain of Thought" (CoT) strategy. Crucially, it incorporates a low-light enhancement algorithm (Scrnet) to clarify environmental images before they are processed by the VLM, ensuring robust performance in dark environments.
Bionic Hybrid Gripper Design
The physical component of GAgent is a specialized bionic hybrid gripper inspired by the anatomy of the human hand. The design utilizes a combination of materials to achieve variable stiffness:
- Bones: Rectangular springs provide the internal structural framework.
- Flesh: A silicone outer layer ensures a soft, conforming interface for delicate objects.
- Tendons: Motor-driven cables allow the system to apply pretension. By pulling these cables, the gripper can dynamically increase its stiffness, transitioning from a soft state to a more rigid configuration.
Experimental Evaluation and Key Insights
GAgent was evaluated using an open-world benchmark consisting of 42 diverse items, including daily necessities and industrial tools such as eggs, seaweed, hammers, and power clips. Tests were conducted across three lighting scenarios: normal daylight, noon overexposure, and evening low light.
Cognition-Physique Feedback Loop
A key insight from our research is the effectiveness of the feedback loop between the VLM and the hardware. The VLM does not merely detect an object; it infers material properties. For instance, when identifying a fragile peach, the VLM commands the gripper to adopt a low stiffness level. In cases of initial failure—such as misidentifying a hard durian shell as a soft fruit—the system's reasoning loop allows for self-correction, re-evaluating the material and adjusting stiffness for a successful second attempt.
Dynamic Stability
Beyond load capacity, the variable-stiffness mechanism significantly improves the gripper's dynamic stability. Pre-stretching the internal springs dampens vibrations, which is critical for drone-mounted applications. Experimental data showed that GAgent stabilized 28% faster (1.8s vs. 2.5s) and reduced vibration amplitude from 25° to 19° compared to non-pretensioned states.
Results and Conclusion
GAgent successfully grasped all 42 items in the benchmark, demonstrating exceptional generalization without the need for task-specific training data. The integration of the brightness adjustment tool proved vital; without it, the system failed to focus on objects in extreme lighting, whereas the enhanced workflow maintained high success rates. The hybrid design allows the agent to lift loads up to 700g while remaining gentle enough to handle delicate items like potato chips.
Future work will focus on integrating GAgent onto Unmanned Aerial Vehicles (UAVs) for active flight testing in extreme outdoor conditions and optimizing smaller, local models to reduce the latency currently associated with high-level VLMs like GPT-4.
Summary updated