DaDu-E: Rethinking the Role of Large Language Model in Robotic Computing Pipeline

Wenhao Sun, Sai Hou, Zixuan Wang, Bo Yu, Shaoshan Liu, Xu Yang, Shuai Liang, Yiming Gan, Yinhe HanarXiv 2024 · DOI · arXiv

Introduction

The current landscape of embodied AI is dominated by "monolithic" Large Language Models (LLMs) like GPT-4o or PaLM-E. These models, while impressive in their reasoning capabilities, present significant hurdles for real-world robotics. Their massive parameter counts necessitate cloud-based execution, introducing latency and privacy risks. Moreover, many existing robotic planners operate in an open-loop fashion, where the LLM generates a plan that the robot executes without the ability to adapt to failures or environmental changes.

DaDu-E (Dalu-Duer) addresses these challenges by proposing a robust, closed-loop robotic planning framework. It demonstrates that high-level intelligence in robotics can be achieved using lightweight, local models—specifically Llama 3.1-8B—when paired with a well-designed computing pipeline featuring visual feedback and memory augmentation.

The DaDu-E Architecture

The core philosophy of DaDu-E is to move away from brute-force scaling. Instead, it utilizes a modular architecture that integrates high-level task planning with low-level action execution through a feedback-rich loop.

The DaDu-E architecture featuring high-level planning, environment perception, and low-level action APIs.

Lean Instruction Sets and Action APIs

To enable a smaller model like Llama 3.1-8B to reason effectively, the researchers developed a minimal set of "Action APIs": navigate, pick, and place. By encapsulating complex low-level controls into these discrete functions, the prompt context length is reduced to an average of 543 tokens. This efficiency allows the local LLM to focus on high-level logic rather than getting bogged down in the minutiae of joint control.

Closed-Loop Feedback and VLM Integration

Unlike open-loop systems, DaDu-E incorporates a Vision-Language Model (VLM), LLaVA-OneVision-8B, to provide real-time visual verification. If a robot attempts to pick up an object but fails, or if it navigates to the wrong side of a table, the VLM identifies the discrepancy. This feedback is fed back into the LLM, triggering a "replan" phase. This closed-loop mechanism ensures the robot can recover from errors autonomously.

Memory Augmentation

DaDu-E employs a dual-memory system to enhance efficiency:

  • Short-term (Episodic) Memory: Tracks the status and location of objects encountered during the current task.
  • Long-term (Semantic) Memory: Stores persistent environmental data, such as 2D semantic maps and obstacle locations. This memory module prevents redundant exploration, allowing the robot to "remember" where a target was last seen rather than searching the entire room again.

Experimental Setup and Physical Implementation

The framework was validated in both Gazebo simulations and a real-world environment. The physical platform consists of a Dalu mobile base and a UR3 robotic arm equipped with a Realsense depth camera and LiDAR.

The real-world DaDu-E robot setup featuring the UR3 arm and Dalu base.

The researchers established a four-level benchmark to test the system's limits:

  1. Basic Execution: Single-step grasp and place.
  2. Multi-Task: Sequential operations (e.g., "Move the apple to the fruit table and the water to the drinks table").
  3. Autonomous Decision-Making: Tasks requiring comparative reasoning (e.g., "Find the largest fruit").
  4. High-Level Cognition: Interpreting vague human requests (e.g., "I'm hungry").

Performance and Key Results

DaDu-E's performance was compared against state-of-the-art baselines, including COME-Robot (GPT-4o based) and Code as Policies (CaP).

Computational Efficiency

The most striking result is the efficiency gain. DaDu-E requires only 26,064 GFLOPs for inference, a staggering reduction compared to the 1.2 million GFLOPs required by GPT-3.5-based systems. It achieves this while using only 0.008× the parameters of the COME-Robot planner.

Success Rates and Navigation

In Level 1, 2, and 4 tasks, DaDu-E matched the success rates of the much larger GPT-4o based systems (0.90, 0.70, and 0.95 respectively). Ablation studies highlighted the importance of the feedback loop; without it, success rates in complex multi-task scenarios dropped by up to 12.7×. Furthermore, the memory module reduced system latency by 34.5% in multi-object tasks by eliminating redundant navigation.

Discussion and Future Work

While DaDu-E excels at structured tasks, it still lags behind frontier models in Level 3 (Autonomous Decision-Making), where it achieved a 0.51 success rate compared to GPT-4o's 0.85. This suggests that while 8B-parameter models are highly capable planners, they still face limitations in open-ended, comparative reasoning.

Future developments will focus on tighter integration between the VLM and the memory module to improve performance in dynamic, previously unseen environments. The success of DaDu-E serves as a blueprint for deploying sophisticated embodied AI on local hardware, prioritizing pipeline intelligence over model size.

Summary updated

BibTeX
@misc{sun2024dadu,
  author = {Wenhao Sun and Sai Hou and Zixuan Wang and Bo Yu and Shaoshan Liu and Xu Yang and Shuai Liang and Yiming Gan and Yinhe Han},
  title = {{DaDu-E: Rethinking the Role of Large Language Model in Robotic Computing Pipeline}},
  year = {2024},
  eprint = {2412.01663},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.01663},
}
Figure