What Can a VLA Learn from the Agent That Calls It? Task-Level Experience Distillation for Embodied Navigation

Jinze Tao, Yuxiao Han, Kangkang Dong, Yancong Wei, Xiaojun Zhu, Bokui Chen, Houde Liu

This paper introduces H2E (Harness-to-Executor), a task-level experience distillation method that significantly improves the performance and efficiency of Vision-Language-Action (VLA) models for embodied navigation. By training an 8B parameter VLA on successful sessions from a high-level LLM-based planner, the authors enable the VLA to navigate autonomously at test time, achieving a 9.9-point success rate boost while running nearly 4 times faster than the original agent.

TL;DR

H2E distills the complex, multi-step reasoning of an expensive LLM-guided navigation agent into a standalone VLA model. By relabeling agent trajectories with task-level goals and masking out unproductive "detours," the distilled VLA learns to find distant objects and stop accurately without needing any LLM calls at test time.

What Problem Does This Paper Address?

State-of-the-art embodied navigation often relies on a hierarchical "brain and muscles" approach. A "frontier" Large Language Model (LLM) acts as a planner, maintaining maps and issuing sub-goals to a Vision-Language-Action (VLA) model that handles low-level execution.

While effective, this is incredibly inefficient. A single navigation episode can consume upwards of 70,000 LLM tokens and take twice as long as a standalone model due to constant communication overhead. Previous research has tried to distill the planner into a smaller model, but H2E takes a different route: making the executor smart enough to work alone.

The challenge is that agent sessions are messy. They include motions driven by internal planner tools, sub-instructions the VLA won't see at test time, and "detours" where the agent makes a mistake and then corrects itself. Standard imitation learning fails on this noisy data.

Methods

The authors propose H2E (Task-Level Experience Distillation), which transforms successful agent sessions into clean training data for the VLA using three core rules:

  1. Task-Level Re-conditioning: Every motion in a successful session is relabeled as a VLA action under the original high-level instruction (e.g., "Go to the kitchen"), regardless of whether the motion was originally triggered by a planner tool or a specific sub-instruction.
  2. Progress Masking: Navigation is rarely a straight line. H2E identifies "detours"—steps that move away from the goal—and masks their contribution to the loss function. This ensures the model only learns from progress-making actions while still seeing the full visual history.
  3. The Stop Rule: Base VLAs often struggle with knowing when to stop, frequently overshooting the goal. H2E labels every state visited within 1 meter of the goal as a "stop" action.
The H2E pipeline: (a) collecting successful sessions from an LLM-VLA harness, (b) converting sessions to labels via re-conditioning and masking, and (c) distilling into the executor using LoRA.

The model is fine-tuned using Low-Rank Adaptation (LoRA) on an 8B NavGPT VLA. A key mathematical insight in the paper is the use of an $L_1$ loss; the authors prove that under certain conditions, $L_1$ loss naturally drives the model to output an exact zero (a stop) when near the goal, whereas a squared loss would merely output a "mean" action that keeps the robot moving.

Datasets

The primary benchmark used is REVERIE-CE (Remote Embodied Visual Referring Expression in Continuous Environments).

  • Training: 2,395 successful sessions collected from 57 training scenes.
  • Evaluation: The full "val-unseen" split, consisting of 3,189 episodes across 10 scenes that the model never encountered during training.

Key Insights and Ideas

  • Learning from the "Harness," not just the model: The VLA doesn't just learn to imitate itself; it learns from the tools the planner used (like "walk back to a previous node"). This allows the VLA to inherit high-level strategies it couldn't discover on its own.
  • The Power of Masking: Nearly a quarter of the steps in agent sessions move away from the goal. Without progress masking, the VLA would learn to replicate these mistakes.
  • Distillation as Efficiency: By moving the "intelligence" from the LLM planner into the VLA's weights, the system eliminates the need for 70k tokens per episode and the latency of API calls.

Key Results

On the REVERIE-CE val-unseen benchmark, H2E delivers impressive gains:

  • Success Rate (SR): Jumps from 35.0% to 44.9%, a nearly 10-point absolute improvement.
  • Efficiency: The distilled model runs 3.7x faster than the full LLM-VLA agent and uses zero LLM tokens at test time.
  • Better Stopping: As shown in Figure 5, the base VLA often walks right through the goal region and stops outside it. H2E learns to "stick the landing" inside the 3-meter success zone.
Qualitative comparison showing H2E stopping accurately within the goal region while the base VLA overshoots.
  • Finding New Goals: H2E reached the goal in 401 episodes that the base VLA never even entered, proving it learned better global navigation strategies from the planner.

Limitations and Future Work

  • Recovery: While H2E is better at reaching the goal, it hasn't yet learned how to recover if it goes significantly off-track—a skill the original LLM planner possesses because it has access to a persistent map and memory.
  • Scope: The study focused on one specific benchmark and one VLA architecture. Testing the generality of H2E across different robot embodiments and tasks is a natural next step.
  • Failure Analysis: The authors suggest that training on sessions that started from VLA failures could further improve robustness.

Summary updated

Abstract

Navigation agents use a large language model (LLM) planner to call a vision-language-action (VLA) model. Each episode needs the LLM, adding cost and delay. We distill successful agent experience into the VLA instead of distilling the planner. The VLA then navigates alone at test time. Agent sessions are not VLA demonstrations. They include motions driven by planner tools and sub-instructions, along with detours. We present H2E, a task-level experience distillation method that converts these sessions into VLA training data. Every motion becomes a VLA action under the original task instruction, with the resulting poses as targets. H2E masks the loss on steps that move away from the goal. It labels every visited state within 1 m of the goal as a stop. We then update the VLA with low-rank adaptation. The VLA learns to stop at the goal and find goals the base model never reaches. On all 3,189 REVERIE-CE val-unseen episodes, H2E raises the success rate of an 8B VLA from 35.0% to 44.9%.
BibTeX
@misc{tao2026what,
  author = {Jinze Tao and Yuxiao Han and Kangkang Dong and Yancong Wei and Xiaojun Zhu and Bokui Chen and Houde Liu},
  title = {{What Can a VLA Learn from the Agent That Calls It? Task-Level Experience Distillation for Embodied Navigation}},
  year = {2026},
  eprint = {2611.16246},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16246},
}
Figure