Introduction
The current landscape of embodied AI is dominated by "monolithic" Large Language Models (LLMs) like GPT-4o or PaLM-E. These models, while impressive in their reasoning capabilities, present significant hurdles for real-world robotics. Their massive parameter counts necessitate cloud-based execution, introducing latency and privacy risks. Moreover, many existing robotic planners operate in an open-loop fashion, where the LLM generates a plan that the robot executes without the ability to adapt to failures or environmental changes.
DaDu-E (Dalu-Duer) addresses these challenges by proposing a robust, closed-loop robotic planning framework. It demonstrates that high-level intelligence in robotics can be achieved using lightweight, local models—specifically Llama 3.1-8B—when paired with a well-designed computing pipeline featuring visual feedback and memory augmentation.
The DaDu-E Architecture
The core philosophy of DaDu-E is to move away from brute-force scaling. Instead, it utilizes a modular architecture that integrates high-level task planning with low-level action execution through a feedback-rich loop.
Lean Instruction Sets and Action APIs
To enable a smaller model like Llama 3.1-8B to reason effectively, the researchers developed a minimal set of "Action APIs": navigate, pick, and place. By encapsulating complex low-level controls into these discrete functions, the prompt context length is reduced to an average of 543 tokens. This efficiency allows the local LLM to focus on high-level logic rather than getting bogged down in the minutiae of joint control.
Closed-Loop Feedback and VLM Integration
Unlike open-loop systems, DaDu-E incorporates a Vision-Language Model (VLM), LLaVA-OneVision-8B, to provide real-time visual verification. If a robot attempts to pick up an object but fails, or if it navigates to the wrong side of a table, the VLM identifies the discrepancy. This feedback is fed back into the LLM, triggering a "replan" phase. This closed-loop mechanism ensures the robot can recover from errors autonomously.
Memory Augmentation
DaDu-E employs a dual-memory system to enhance efficiency:
- Short-term (Episodic) Memory: Tracks the status and location of objects encountered during the current task.
- Long-term (Semantic) Memory: Stores persistent environmental data, such as 2D semantic maps and obstacle locations. This memory module prevents redundant exploration, allowing the robot to "remember" where a target was last seen rather than searching the entire room again.
Experimental Setup and Physical Implementation
The framework was validated in both Gazebo simulations and a real-world environment. The physical platform consists of a Dalu mobile base and a UR3 robotic arm equipped with a Realsense depth camera and LiDAR.
The researchers established a four-level benchmark to test the system's limits:
- Basic Execution: Single-step grasp and place.
- Multi-Task: Sequential operations (e.g., "Move the apple to the fruit table and the water to the drinks table").
- Autonomous Decision-Making: Tasks requiring comparative reasoning (e.g., "Find the largest fruit").
- High-Level Cognition: Interpreting vague human requests (e.g., "I'm hungry").
Performance and Key Results
DaDu-E's performance was compared against state-of-the-art baselines, including COME-Robot (GPT-4o based) and Code as Policies (CaP).
Computational Efficiency
The most striking result is the efficiency gain. DaDu-E requires only 26,064 GFLOPs for inference, a staggering reduction compared to the 1.2 million GFLOPs required by GPT-3.5-based systems. It achieves this while using only 0.008× the parameters of the COME-Robot planner.
Success Rates and Navigation
In Level 1, 2, and 4 tasks, DaDu-E matched the success rates of the much larger GPT-4o based systems (0.90, 0.70, and 0.95 respectively). Ablation studies highlighted the importance of the feedback loop; without it, success rates in complex multi-task scenarios dropped by up to 12.7×. Furthermore, the memory module reduced system latency by 34.5% in multi-object tasks by eliminating redundant navigation.
Discussion and Future Work
While DaDu-E excels at structured tasks, it still lags behind frontier models in Level 3 (Autonomous Decision-Making), where it achieved a 0.51 success rate compared to GPT-4o's 0.85. This suggests that while 8B-parameter models are highly capable planners, they still face limitations in open-ended, comparative reasoning.
Future developments will focus on tighter integration between the VLM and the memory module to improve performance in dynamic, previously unseen environments. The success of DaDu-E serves as a blueprint for deploying sophisticated embodied AI on local hardware, prioritizing pipeline intelligence over model size.
Summary updated