TL;DR
RoboCerebra introduces a new large-scale benchmark designed to push robotic manipulation away from purely reactive, short-horizon actions ("System 1") toward deliberative, long-horizon reasoning ("System 2"). Featuring human-annotated trajectories that are roughly 6× longer than existing datasets, it provides a rigorous framework for evaluating how well vision-language models (VLMs) can plan, remember, and adapt in dynamic environments.
What Problem Does This Paper Address?
Most state-of-the-art vision-language-action (VLA) models operate as "System 1" controllers: they map immediate visual inputs directly to low-level motor actions. While highly effective for short, reactive tasks (like picking up a single object), they struggle with complex, multi-step household chores that require long-term planning, memory, and error recovery.
Existing benchmarks typically feature short tasks (fewer than 500 action steps) and assume static, fully observable environments. They fail to test "System 2" cognitive capabilities—such as tracking hidden states (remembering what is inside a closed microwave), actively exploring an environment to find an item, or re-planning after an unexpected physical disturbance. RoboCerebra fills this gap by providing a benchmark explicitly designed to test these high-level cognitive skills over extended temporal horizons.
Methods
The authors tackle long-horizon manipulation through two main contributions:
- Hierarchical Planning and Execution (HPE) Framework: Instead of relying on a single end-to-end model, the authors split the labor. A high-level System 2 VLM Planner processes low-frequency observations to generate and update step-level subgoals stored in a memory bank. Concurrently, a low-level System 1 VLA Controller (built on OpenVLA) consumes high-frequency visual inputs to execute the fine-grained motor skills required to fulfill those subgoals.
- Automated & Verified Data Generation Pipeline: To scale up dataset creation, the authors developed a top-down pipeline. An LLM (GPT-o3-mini) generates diverse high-level tasks and decomposes them into subtasks based on available objects. This plan is converted into simulator code, which undergoes a dual-loop verification process (symbolic checks + visual verification via GPT-4o) to ensure physical and semantic plausibility. Finally, human operators execute the subtasks in simulation to collect high-quality trajectories.
Datasets
The RoboCerebra dataset is built on the Libero simulation platform and consists of:
- 1,000 human-annotated trajectories spanning 100 distinct task variants in household environments.
- An average trajectory length of 2,972.4 simulation steps per task—roughly 6× longer than previous benchmarks like VLABench or RoboCasa.
- Tasks categorized into 6 distinct settings to isolate specific cognitive demands: Ideal (static/fully observable), Memory Exploration, Memory Execution, Random Disturbance, Observation Mismatching, and Mix (combining memory and dynamic factors).
Key Insights and Ideas
- System 1 alone is blind to time: Pure VLA models fail dramatically on long-horizon tasks. Even when fine-tuned, a standalone OpenVLA model achieved a success rate of under 8% in ideal settings and 0% in complex "Mix" settings, proving it cannot maintain instruction fidelity over long periods.
- The System 2 Boost: Introducing a high-level VLM planner dramatically improves success rates in complex, dynamic, and memory-intensive environments by breaking down long-term dependencies into manageable subgoals.
- The Complexity Penalty: Interestingly, in simple, fully observable ("Ideal") tasks, the Hierarchical Framework performed slightly worse than a pure planner baseline (a 0.82% drop). This suggests that unnecessary reasoning overhead can sometimes introduce suboptimal decision-making in straightforward scenarios.
- Generalists vs. Specialists: While specialized embodied VLMs (like RoboBrain-2.0) excel at environment QA and reflection, they do not yet fully generalize to embodied planning. General-purpose models like GPT-4o still lead the pack in overall planning accuracy and efficiency.
Key Results
- Trajectory Length: RoboCerebra features an average of 2,972.4 steps per task, compared to VLABench (502.5 steps) and RoboCasa (371.9 steps).
- Framework Performance: In the challenging Mix setting, the fine-tuned OpenVLA baseline scored 0.00%, whereas the Hierarchical Framework achieved a 13.21% success rate.
- Planner Evaluation: Among the tested System 2 models, GPT-4o achieved the highest average success rate (16.04%) and the highest planning accuracy (68.33%), outperforming specialized models and other open-source VLMs like Qwen2.5-VL.
Limitations and Future Work
The authors note that the current interaction between System 1 and System 2 is relatively rigid and primarily one-way (the planner dictates to the controller). Future work will focus on developing richer bidirectional communication, allowing the low-level controller to provide fine-grained, interpretable feedback to the planner for better failure recovery. Additionally, extending the benchmark to real-world robotic platforms will be crucial to validating these System 2 capabilities outside of simulation.
Summary updated