Abstract
Bridging the gap between high-level abstract human instructions and low-level robotic execution remains a significant challenge in robotics. This paper introduces a novel framework that leverages the reasoning capabilities of Large Language Models (LLMs), specifically GPT-4, combined with Retrieval-Augmented Generation (RAG) and real-time Integrated Force and Visual Feedback (IFVF). This architecture allows robots to translate vague commands into precise, long-horizon physical actions while maintaining robustness in dynamic, unpredictable environments. We demonstrate the system's efficacy through complex tasks such as beverage preparation and artistic plate decoration, highlighting the critical role of force feedback in achieving physical grounding.
Introduction
While LLMs have demonstrated remarkable proficiency in high-level planning, their application in the physical world is often hindered by a lack of "physical grounding." Most existing robotic systems struggle with long-horizon tasks where errors in early sub-steps cascade, and they frequently fail in dynamic environments where objects may be moved during execution. To address these issues, a robot must not only plan but also perceive and react to physical stimuli in real-time.
As shown in Figure 1, the proposed system is tested in a realistic kitchen-like environment. Tasks such as opening drawers, scooping coffee, and pouring water require the robot to handle various objects and respond to physical disturbances, such as a human moving a mug mid-pour.
Methodology: The IFVF Framework
The core of our approach is a multi-layered architecture that treats "Code as Policies." The system decomposes high-level reasoning from low-level control through a sophisticated pipeline.
High-Level Reasoning and RAG
When a user provides a voice query, GPT-4 analyzes the command alongside visual data from the environment to decompose the task into actionable sub-goals. To ensure scalability and precision, we utilize Retrieval-Augmented Generation (RAG). Instead of embedding all possible robotic skills into the LLM's context, the system queries a curated Knowledge Base for relevant Python code snippets (motion primitives). This allows the model to generate task-specific code that is both efficient and grounded in validated robotic motions.
Integrated Force and Visual Feedback (IFVF)
The "secret sauce" of this framework is the integration of real-time feedback loops directly into the generated code. As illustrated in Figure 2, the system processes visual data ($s_{visual}$) from an Azure Kinect camera to track object poses and high-frequency force data ($s_{force}$) from an ATI multi-axis sensor.
Force feedback is particularly critical for tasks involving occlusion. For instance, when pouring water, the kettle often blocks the camera's view of the mug. By monitoring the weight change in real-time, the robot can achieve precision that vision-only systems cannot match. Furthermore, these feedback loops allow for reactive adaptation; if a human moves an object, the robot detects the change in state and adjusts its trajectory accordingly.
Experimental Results
The framework was evaluated on two primary benchmarks: a multi-step coffee-making sequence and a plate-decoration task.
Pouring and Manipulation
In pouring tasks, the robot achieved an accuracy of approximately 5.4 grams per 100 grams at controlled speeds. The system also demonstrated the ability to open drawers with unknown mechanisms by sensing resistance and adjusting its pull vector, showcasing the importance of force-sensing in interacting with the physical world.
Creative Autonomy
By linking generative models like DALL-E to the robot's trajectory planner, the system performed artistic tasks. In plate decoration, the robot achieved a Jaccard Index completeness score of 98-99% across various categories. The force control was essential here to maintain even pen pressure on the uneven surfaces of ceramic plates.
Discussion and Limitations
While the framework represents a significant step toward autonomous domestic robots, several limitations remain. The current system is primarily reactive; it adjusts to changes but does not yet proactively re-plan entire task sequences without new user input. Additionally, vision latency (0.33 Hz) limits performance in high-speed scenarios. Future work will focus on integrating more sophisticated fluid dynamics for pouring and improving the temporal resolution of visual perception.
Conclusion
By combining the cognitive power of LLMs with the physical sensitivity of force and visual feedback, we have developed a system capable of following abstract instructions in messy, real-world environments. This integration of reasoning and sensing is vital for the next generation of collaborative and domestic robots.
Summary updated