FrankenBot: Brain-Morphic Modular Orchestration for Robotic Manipulation with Vision-Language Models

Shiyi Wang, Wenbo Li, Yiteng Chen, Qingyao Wu, Huiping ZhuangarXiv 2025 · DOI · arXiv

FrankenBot introduces a brain-inspired, modular framework for robotic manipulation that leverages Vision-Language Models (VLMs) without the need for additional fine-tuning. By mapping core functions like task planning, anomaly handling, and memory to distinct "brain" modules, it achieves comprehensive robotic capabilities with high operational efficiency—typically requiring just a single VLM call per task.

The Challenge of VLM-Driven Robotics

Vision-Language Models (VLMs) possess incredible scene understanding and reasoning capabilities, making them highly attractive for robotic task planning. However, current VLM-driven robotic systems usually focus on just one piece of the puzzle—like planning or error correction.

If discrete VLM functions are naively stitched together to build a complete "robotic brain," the system performance degrades significantly. Because each VLM call takes several seconds, a robot constantly pinging a cloud model for every step or minor error becomes practically useless in dynamic environments. Alternatively, end-to-end Vision-Language-Action (VLA) models exist, but they require massive amounts of cross-modal training data and lack plug-and-play flexibility. FrankenBot addresses this core challenge: building a fully functional, zero-shot robotic brain that minimizes expensive VLM calls while maintaining robust error correction and memory.

Methodology: A Brain-Morphic Framework

The authors propose FrankenBot, a framework that mimics the neurobiological structure of the human brain to decouple high-frequency control from low-frequency VLM reasoning.

The brain-morphic architecture of FrankenBot, mapping VLM functions to biological brain regions.

The architecture is divided into several key modules:

  • Cortex (VLM): Handles high-level task planning and reasoning.
  • Cerebellum (HELLO Module): Manages execution control and anomaly handling.
  • Hippocampus (HIMM Module): Manages short, medium, and long-term memory.
  • Brainstem: Handles low-level hardware interfaces.

To achieve efficiency, FrankenBot employs several mechanisms:

  1. HiErarchicaL dynamic feedback Local cOntrol (HELLO): Instead of step-by-step VLM queries, the VLM generates a Turing-complete Hierarchical Execution Tree (HET) upfront. This tree includes loops, conditionals, and atomic operations, allowing autonomous execution of complex logic.
  2. Multi-level Anomaly Handling (MAH): A 3-tier error correction system. The VLM generates predefined monitors to catch simple errors instantly. If that fails, a Local Anomaly Expert (a lightweight local LLM) attempts to fix the sequence. Only highly complex errors escalate to a full VLM Replan.
  3. Hierarchical Incremental Memory Management (HIMM): A tiered memory system that caches logs and uses a local LLM to summarize frequently used VLM-generated code into an Incremental Skill Pool. This allows the robot to reuse code for common tasks rather than regenerating it.

Experimental Evaluation

As a plug-and-play, zero-shot framework, FrankenBot does not require offline datasets for training. For evaluation, the authors designed 10 open-vocabulary real-world manipulation tasks, including Block Stacking, Drawer Closing, Tea Pouring, and USB Plugging.

Four of the ten real-world manipulation tasks used to evaluate FrankenBot.

The physical setup utilized a UR5e 6-DoF robotic arm and an Intel RealSense D435i RGB-D camera. FrankenBot was compared against state-of-the-art baselines VoxPoser and ReKep.

Core Insights

  • The "Regeneration Trap" Bottleneck: Up to 48% of VLM outputs in robotic tasks are functionally identical but textually varied. Caching these into a reusable skill pool drastically cuts latency and API costs.
  • Skewed Anomaly Distribution: In real-world tests, ~70% of errors are simple or predictable, ~20% are recoverable sequence errors, and only ~10% are complex failures. FrankenBot's 3-tier anomaly handling exploits this, routing 70% of errors to instant local code.
  • Single-Call Efficiency: By generating multi-threaded executable code (both the execution tree and anomaly monitors) in one go, the robot operates autonomously for the majority of a task.

Performance and Efficiency

  • Superior Success Rates: FrankenBot achieved a 73% overall success rate across 10 tasks, significantly outperforming VoxPoser (46%) and ReKep (55%), particularly on long-horizon tasks.
  • Faster Execution: Task completion times plummeted. For example, Drawer Closing took 5.3s (vs. ReKep's 17.1s), and Block Picking took 7.2s (vs. VoxPoser's 12.4s).
  • Efficient Error Correction: The MAH system handled predictable anomalies with a 97% success rate while incurring only a 0.3s time penalty, compared to 11.6s for full VLM replanning.
  • Memory Benefits: Ablation studies showed the HIMM module reduced generated code size by 41% and cut VLM calls by 27%.

Discussion and Future Directions

While FrankenBot shows significant promise, it relies heavily on the reasoning power of top-tier VLMs (e.g., GPT-4), with performance dropping on smaller models. Additionally, fine-grained scene understanding and low-level motion control remain dependent on predefined interfaces.

Future work includes exploring Retrieval-Augmented Generation (RAG) for more robust long-term memory, domain-adaptive fine-tuning for better spatial reasoning, and integrating advanced Vision-Language-Action (VLA) architectures for more graceful low-level control.

Summary updated

Abstract

Developing a general robot manipulation system capable of performing a wide range of tasks in complex, dynamic, and unstructured real-world environments has long been a challenging task. It is widely recognized that achieving human-like efficiency and robustness manipulation requires the robotic brain to integrate a comprehensive set of functions, such as task planning, policy generation, anomaly monitoring and handling, and long-term memory, achieving high-efficiency operation across all functions. Vision-Language Models (VLMs), pretrained on massive multimodal data, have acquired rich world knowledge, exhibiting exceptional scene understanding and multimodal reasoning capabilities. However, existing methods typically focus on realizing only a single function or a subset of functions within the robotic brain, without integrating them into a unified cognitive architecture. Inspired by a divide-and-conquer strategy and the architecture of the human brain, we propose FrankenBot, a VLM-driven, brain-morphic robotic manipulation framework that achieves both comprehensive functionality and high operational efficiency. Our framework includes a suite of components, decoupling a part of key functions from frequent VLM calls, striking an optimal balance between functional completeness and system efficiency. Specifically, we map task planning, policy generation, memory management, and low-level interfacing to the cortex, cerebellum, temporal lobe-hippocampus complex, and brainstem, respectively, and design efficient coordination mechanisms for the modules. We conducted comprehensive experiments in both simulation and real-world robotic environments, demonstrating that our method offers significant advantages in anomaly detection and handling, long-term memory, operational efficiency, and stability -- all without requiring any fine-tuning or retraining.
BibTeX
@misc{wang2025frankenbot,
  author = {Shiyi Wang and Wenbo Li and Yiteng Chen and Qingyao Wu and Huiping Zhuang},
  title = {{FrankenBot: Brain-Morphic Modular Orchestration for Robotic Manipulation with Vision-Language Models}},
  year = {2025},
  eprint = {2506.21627},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.21627},
}
Figure