"Pass the butter": A study on desktop-classic multitasking robotic arm based on advanced YOLOv7 and BERT

Haohua Que, Wenbin Pan, Jie Xu, Hao Luo, Pei Wang, Li ZhangarXiv 2024 · DOI · arXiv

Introduction

The quest to transition AI from "brains in a vat" to physically interactive agents has led to the development of a 6-degree-of-freedom (6-DOF) desktop robotic arm. Inspired by the "Butter Robot" from the series Rick and Morty, this study explores how advanced computer vision and natural language processing can be integrated into affordable, edge-computing hardware to perform everyday tasks like passing a cup or flipping a light switch.

Comparison of the fictional 'Butter Robot' and the real-world prototype.

System Architecture

The researchers developed a multimodal framework that balances heavy computational requirements with real-time robotic control. The system architecture is divided between local edge hardware and specialized processing units to manage latency effectively.

  • Hardware Stack: The physical arm is controlled by an STM32F407VET6 for low-level servo management, while a Jetson Nano B01 handles higher-level tasks. The system utilizes a depth camera for spatial mapping and a microphone array for voice input.
  • Natural Language Processing (NLP): Using OpenAI’s Whisper for speech recognition and BERT for joint intent classification and slot filling, the robot can parse natural language commands (e.g., "Pass me the cup") into structured executable functions like Take(cup, goal=user).
  • Computer Vision (CV): Real-time object detection is powered by YOLOv7. To ensure efficiency on the Jetson Nano, the model underwent pruning and quantization. By combining 2D bounding boxes with depth data, the robot calculates precise 3D coordinates for manipulation.
  • Hybrid Offloading: High-latency tasks like NLP intent analysis are offloaded to a remote computer, while high-frequency tasks like CV and motor control remain local to ensure responsiveness.
Hardware specifications and control system layout.

Experimental Validation

The system was evaluated through 200 trials per task across three primary scenarios: operating a door switch, a light switch, and passing a cup. These experiments tested the robot's resilience against varying lighting conditions and background noise levels (ranging from 0% to 75%).

Results and Discussion

The study revealed several key insights into the current state of edge-based robotics:

  • Task Performance: The robot achieved a 92.6% Correct Speech Recognition (CSR) rate for door tasks. However, performance dropped for more complex requests; vague commands like "I'm thirsty" resulted in a 63.4% success rate compared to 81.2% for precise commands.
  • Environmental Sensitivity: Success rates for the light switch task dropped from 84.2% in bright light to 76.6% in dim light, highlighting the dependency of "intelligent" vision on ambient conditions.
  • Hardware Efficiency: Optimized software allowed the budget-friendly Jetson Nano to perform comparably to more expensive hardware like the Orin AGX for these specific tasks.
  • The Execution Bottleneck: While speech recognition remained stable even in noisy environments, physical performance (CP) often lagged due to noise or obstructions, suggesting that the primary challenge is shifting from understanding intent to reliable physical execution.

Conclusion

This research demonstrates a viable path for bringing multimodal AI to small-scale robotics. Future work will focus on improving auditory focus in noisy environments, enabling two-way dialogue for command clarification, and further miniaturizing the system into an all-in-one form factor.

Summary updated

Abstract

In recent years, various intelligent autonomous robots have begun to appear in daily life and production. Desktop-level robots are characterized by their flexible deployment, rapid response, and suitability for light workload environments. In order to meet the current societal demand for service robot technology, this study proposes using a miniaturized desktop-level robot (by ROS) as a carrier, locally deploying a natural language model (NLP-BERT), and integrating visual recognition (CV-YOLO) and speech recognition technology (ASR-Whisper) as inputs to achieve autonomous decision-making and rational action by the desktop robot. Three comprehensive experiments were designed to validate the robotic arm, and the results demonstrate excellent performance using this approach across all three experiments. In Task 1, the execution rates for speech recognition and action performance were 92.6% and 84.3%, respectively. In Task 2, the highest execution rates under the given conditions reached 92.1% and 84.6%, while in Task 3, the highest execution rates were 95.2% and 80.8%, respectively. Therefore, it can be concluded that the proposed solution integrating ASR, NLP, and other technologies on edge devices is feasible and provides a technical and engineering foundation for realizing multimodal desktop-level robots.
BibTeX
@misc{que2024pass,
  author = {Haohua Que and Wenbin Pan and Jie Xu and Hao Luo and Pei Wang and Li Zhang},
  title = {{"Pass the butter": A study on desktop-classic multitasking robotic arm based on advanced YOLOv7 and BERT}},
  year = {2024},
  eprint = {2405.17250},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.17250},
}
Figure