ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learning

Hanyang Chen, Mark Zhao, Rui Yang, Qinwei Ma, Ke Yang, Kangrui Wang, Hao Bai, Zhenhailong Wang, Jiarui Yao, Rui Pan, Mengchao Zhang, Jose Barreiros, Aykut Onol, ChengXiang Zhai, Heng Ji, Manling Li, Huan Zhang, Tong ZhangSubmission to ICLR 2026 · Rejected

TL;DR

Small vision-language models (VLMs) typically struggle with complex physical and semantic tasks, while massive proprietary models are too compute-heavy and slow for real-time robot control. ERA (Embodied Reasoning Agent) bridges this gap with a two-stage recipe: curating structured prior knowledge (EPL) and refining policies through turn-level online reinforcement learning. Using this framework, a compact 3B model (Qwen2.5-VL-3B) outperforms GPT-4o on both high-level semantic planning and low-level continuous robotic manipulation.

What Problem Does This Paper Address?

While frontier models like Claude-3.5-Sonnet and GPT-4o demonstrate impressive prompt-driven planning, deploying them on robots is computationally prohibitive. Conversely, compact open-weight VLMs (3B–7B) fail almost completely out of the box—Qwen2.5-VL-3B scores 0% on standard embodied benchmarks.

Teaching smaller models to act as embodied agents faces several steep roadblocks:

  1. Scarce and shallow demonstrations: Existing trajectories contain raw action sequences without intermediate rationales or fine-grained spatial descriptions.
  2. Context explosion: Long-horizon tasks cause multi-turn interaction histories to bloat ($O(t)$ growth), overwhelming attention heads and compute budgets.
  3. Credit assignment mismatch: Standard RLHF optimizes models at the token level, creating noisy, high-variance gradients when an agent outputs long multi-step thoughts for a single environmental action.

Methods

ERA tackles these hurdles using a two-stage training framework:

[Curated Data Sources] ──> Stage 1: Embodied Prior Learning (EPL)
                                  │
                                  ▼
                            Stage 2: Online Reinforcement Learning
                                  ├── Self-Summarization Context (O(1))
                                  ├── Dense Composite Rewards
                                  └── Turn-Level GAE Policy Optimization

Stage 1: Embodied Prior Learning (EPL)

Before policy exploration, the base VLM is fine-tuned on three complementary data priors:

  • Trajectory-Augmented Priors: Standard trajectories are augmented with structured step-by-step reasoning (visual descriptions, history reflections, and forward plans) distilled from GPT-4o for high-level tasks or rule-based oracle perception for low-level manipulation.
  • Environment-Anchored Priors: Task-level grounding signals beyond demonstrations, including masked action modeling and action reordering for planning, alongside 3D coordinate grounding QA for manipulation.
  • External Knowledge Priors: General reasoning datasets (OpenO1-SFT) and synthetic 3D spatial QA (SpaceThinker) to instill generalizable reasoning.

Stage 2: Turn-Level Online Reinforcement Learning

Agents interact directly with the environment via PPO, powered by three core designs:

  • Self-Summarization Context Management: Instead of retaining the entire raw interaction history, the agent's structured reflection at step $t-1$ compresses historical context into a compact 1-step representation, capping history token length at $O(1)$.
  • Dense Composite Reward Shaping: Rewards combine final task success ($r^\text{success}$), intermediate milestone completion ($r^\text{subgoal}$), and behavioral penalties/bonuses ($r^\text{behavior}$) based on execution validity or visual grounding precision.
  • Turn-Level Generalized Advantage Estimation (Turn-Level GAE): Rather than predicting values for each token, the critic estimates a single scalar value per interaction turn $V_\phi(x_t)$. The resulting turn-level advantage $A_t$ is uniformly shared across all tokens generated in that turn's thought-and-action sequence.

Datasets

The authors train and evaluate ERA on the EmbodiedBench benchmark:

  • EB-ALFRED (High-Level Planning): Household simulation in AI2-THOR spanning seven task families (e.g., Clean & Place, Heat & Place) parameterized via 8 high-level action primitives.
  • EB-Manipulation (Low-Level Control): 7-DoF Franka Emika robotic arm tasks in CoppeliaSim/V-REP requiring continuous 7D end-effector poses.
  • Splits: Evaluated on seen subsets (Base, Complex, Visual) and held-out unseen subsets (Common Sense, Spatial) to test out-of-distribution transfer.
  • Prior Training Data: ALFRED trajectories, VLMbench manipulation trajectories, OpenO1-SFT (10k samples), and SpaceThinker (11.4k samples).

Key Insights and Ideas

  • Turn-level credit assignment stabilizes embodied RL: Treating an entire multi-sentence rationale and action as a single atomic decision radically cuts advantage variance. Switching from token-level to turn-level GAE produced massive leaps: +12.0% on EB-ALFRED and +8.4% on EB-Manipulation.
  • Self-summarization outperforms full history: A single-step context containing the model's own structured reflection achieved higher success rates (47% vs. 45%) with 65% fewer tokens compared to a raw 5-step history, avoiding the distracting noise of context bloat.
  • Different priors play complementary roles: Trajectory augmentation drives the largest individual generalization gains (+13–14% on unseen splits), while environment-anchored data provides localized spatial/action grounding. Combining both delivers the strongest overall performance.
  • Dense rewards scale with horizon: Composite reward shaping yielded a +14% jump on long-horizon planning (EB-ALFRED), but only +2.1% on shorter-horizon manipulation, demonstrating that dense guidance is particularly vital when credit assignment spans dozens of steps.

Key Results

Starting from a baseline of 0% success:

  • EB-ALFRED: ERA-3B achieves a 65.2% average success rate, easily surpassing GPT-4o (56.8%), beating open RL baselines like VAGEN-3B (52.8%), and closely trailing Claude-3.5-Sonnet (66.4%). On unseen spatial tasks, it hits 66.0% (vs. VAGEN's 28.0%).
  • EB-Manipulation: ERA-3B achieves 48.3%, outperforming GPT-4o (28.9%) by +19.4% and VAGEN-3B (22.9%) by +25.4%.
  • Sim-to-Real Transfer: Without any real-world fine-tuning or recalibration, policies transferred zero-shot to a physical 6-DoF XArm6 robot performing pick-and-place tasks.

Limitations and Future Work

  • Strategic Rigidity: Agents occasionally demonstrate conservative exploration—persisting with initial failed search plans rather than dynamically pivoting when an object is missing.
  • Reasoning–Action Decoupling: In rare failure modes, verbalized plans diverge from output actions (e.g., articulating a need for a cup but issuing a command for a bottle), pointing to remaining training distribution bias.
  • Simulation Reliance: While zero-shot deployment was demonstrated on an XArm6, extensive testing was predominantly conducted in simulated environments; full real-world multi-task benchmarking remains open.

Summary updated

Abstract

Recent advances in embodied AI highlight the potential of vision language models (VLMs) as agents capable of perception, reasoning, and interaction in complex environments. However, top-performing systems rely on large-scale models that are costly to deploy, while smaller VLMs lack the necessary knowledge and skills to succeed. To bridge this gap, we present $\textit{Embodied Reasoning Agent (ERA)}$, a two-stage framework that integrates prior knowledge learning and online reinforcement learning (RL). The first stage, $\textit{Embodied Prior Learning}$, distills foundational knowledge from three types of data: (1) Trajectory-Augmented Priors, which enrich existing trajectory data with structured reasoning generated by stronger models; (2) Environment-Anchored Priors, which provide in-environment knowledge and grounding supervision; and (3) External Knowledge Priors, which transfer general knowledge from out-of-environment datasets. In the second stage, we develop an online RL pipeline that builds on these priors to further enhance agent performance. To overcome the inherent challenges in agent RL, including long horizons, sparse rewards, and training instability, we introduce three key designs: self-summarization for context management, dense reward shaping, and turn-level policy optimization. Extensive experiments on both high-level planning (EB-ALFRED) and low-level control (EB-Manipulation) tasks demonstrate that ERA-3B surpasses both prompting-based large models and previous training-based baselines. Specifically, it achieves overall improvements of 8.4\% on EB-ALFRED and 19.4\% on EB-Manipulation over GPT-4o, and exhibits strong generalization to unseen tasks. Detailed Ablation studies further validate the effectiveness of different prior datasets and agent RL designs. Overall, ERA offers a practical path toward scalable embodied intelligence, providing methodological insights for future embodied AI systems.
BibTeX
@misc{chen2025era,
  author = {Hanyang Chen and Mark Zhao and Rui Yang and Qinwei Ma and Ke Yang and Kangrui Wang and Hao Bai and Zhenhailong Wang and Jiarui Yao and Rui Pan and Mengchao Zhang and Jose Barreiros and Aykut Onol and ChengXiang Zhai and Heng Ji and Manling Li and Huan Zhang and Tong Zhang},
  title = {{ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learning}},
  year = {2025},
  eprint = {2510.12693},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.12693},
}
Figure