Improve How You Improve: Recursive Policy Self-Improvement for Zero-Shot Open-Vocabulary Multi-Object Navigation

Yang Li, Ziju Han, Aming Wu, Zihao Zhang, Sijia Zhang, Yahong Han, Qinghua HuarXiv 2026 · arXiv

TL;DR

RPSI-Nav is a recursive self-improvement framework that enables robots to navigate unfamiliar environments more efficiently by refining both their navigation policies and the very logic used to improve them. By closing the loop between navigation outcomes and the "improver" program, the system autonomously diagnoses failures, requests diagnostic observations, and evolves its own repair mechanisms across successive goals.

What Problem Does This Paper Address?

Zero-Shot Object Navigation (ZSON) requires a robot to find objects in a new environment without any prior training on that specific space. In multi-object scenarios, a smart agent should use experience from finding the first object (e.g., a lamp) to find the second (e.g., a pillow) more quickly.

While recent "self-improving" agents can reflect on past trajectories to revise their navigation rules, they typically rely on human-designed, static improvement mechanisms. If the logic used to diagnose a failure is incomplete—for example, if it fixes where the robot looks but ignores how it looks—the agent's ability to improve eventually plateaus. The authors identify a need for "recursive" improvement: the robot shouldn't just improve its navigation; it should improve its ability to improve.

Figure 1. Recursive improvement across goal searches. (a) 𝑆0 finds the lamp and pillow but leaves the plant hypothesis untested. (b) 𝑆2 selects a view that observes the lamp and plant. (c) Search experience guides navigation revisions, while revision outcomes drive the improver’s self-revision. Accepted improver updates govern subsequent revisions. 𝑆𝑘denotes the navigation program 𝑃𝑘in our method.

Methods

The authors propose RPSI-Nav (Recursive Policy Self-Improvement for Navigation). The system maintains two distinct executable programs:

  1. Navigation Program ($P$): Handles location hypotheses, ranking, and viewpoint selection.
  2. Improvement Program ($R$): Contains rules for diagnosis ($D$), observation acquisition ($Q$), and revision planning ($U$).

The core innovation is the Recursive Loop. When the robot fails or performs inefficiently, the Improvement Program ($R$) analyzes the logs. If the cause of a failure is ambiguous, $R$ can trigger a "diagnostic view"—spending a portion of the movement budget specifically to gather evidence that distinguishes between competing explanations (e.g., "Is my ranking bad?" vs. "Is my viewpoint occluded?").

Crucially, feedback from subsequent goals is used to evaluate the revisions themselves. If a previous fix to the navigation program ($P$) proves ineffective, the system uses this as evidence to revise the Improvement Program ($R$). This creates a hierarchy where the agent evolves its own diagnostic and repair rules over time. To ensure stability, RPSI-Nav uses Conditional Inheritance, where updates are only carried forward if they pass "applicability guards" based on the current context and budget.

Figure 2. Overview of RPSI-Nav. Goals Q, RGB-D observations (𝐼𝑡, 𝐷𝑡), pose 𝜉𝑡, and memory M𝑡support navigation by 𝑃𝑘, yielding actions 𝑎𝑡, trajectory 𝜏, and goal confirmations. A fixed harness executes actions with frozen-VLM perception. 𝑅𝑘= (𝐷𝑘, 𝑄𝑘,𝑈𝑘) guides diagnosis, observation acquisition, and revision through the shared agent workflow. Augmented records B+ and revision feedback support patches Δ𝑋𝑘, 𝑋∈{𝑃, 𝑅}. Accepted patches apply under guard 𝑔𝑘(𝑥). Revision outcomes drive 𝑅𝑘’s self-revision, and accepted 𝑅𝑘+1 governs subsequent navigation revisions, closing the recursive loop.

Datasets

The framework was evaluated on several challenging benchmarks:

  • HM3D (Habitat Matterport 3D): A standard benchmark for zero-shot multi-object navigation.
  • HM3D-MFMON: A subset specifically designed to test navigation across multiple floors.
  • GOAT-Bench: A benchmark for multimodal navigation involving category names, language descriptions, and reference images.
  • Real-Robot Evaluation: Qualitative tests performed on a quadruped robot in a real indoor environment.

Key Insights and Ideas

  • Meta-Improvement is Key: The "aha" moment is that the logic for fixing code is itself code. By making the improver editable, the system can overcome the limitations of its initial human-designed heuristics.
  • Diagnostic Probing: In embodied AI, information isn't free. RPSI-Nav treats "moving to learn why I failed" as a first-class action, distinct from "moving to find the goal."
  • Long-Horizon Feedback: A failure at the third goal might reveal that a "fix" applied after the first goal was actually flawed. RPSI-Nav explicitly links these long-term dependencies to refine its meta-rules.

Key Results

RPSI-Nav demonstrated significant performance gains over state-of-the-art baselines:

  • HM3D (Sequential Protocol): Improved Success Rate (SR) from 54.20% (OneMap baseline) to 63.14% and Success weighted by Path Length (SPL) from 27.80% to 35.81%.
  • Multi-Floor Navigation: On the Cross-Floor-Required subset of HM3D-MFMON, it achieved 15.97% SR, a massive jump from the 0% SR of the base OneMap system.
  • Ablation Studies: The researchers found that allowing the improver to evolve ($R$ revising $R$) provided a +3.3 SR and +3.7 SPL boost over a version that only allowed the navigation policy to evolve.
Figure 4. Navigation comparison and a representative RSI trace. Upper row: OneMap fails to complete the goal sequence. Lower row: RPSI-Nav completes the chair, toilet, and sofa searches. Top-down maps and RGB views show selected timesteps; boxes annotate visible targets. The three stages illustrate the recursive repair process: (a) 𝑅0 revises region ranking, producing 𝑃1. (b) Repeated views prompt 𝑅0 to review the repair and add a viewpoint-coverage check, yielding 𝑅1. (c) 𝑅1 invokes this check and redirects the next repair to viewpoint selection, producing 𝑃2. 𝑃𝑘and 𝑅𝑘denote the navigation and improvement programs, respectively.

Limitations and Future Work

  • Movement Budget Trade-offs: Diagnostic views are powerful but consume the same movement budget as navigation. Finding the optimal balance between "learning" and "doing" remains an open challenge.
  • LLM Dependency: The system relies on frozen Large Language Models (like Qwen2.5-Coder) for reasoning and code generation. The quality of self-improvement is inherently bounded by the underlying model's coding and logic capabilities.
  • Real-World Dynamics: While tested on a quadruped, the high-level programmatic revisions might need more robust integration with low-level controllers to handle highly dynamic or cluttered real-world environments.

Summary updated

Abstract

To advance the development of embodied general navigation, zero-shot open-vocabulary multi-object navigation has attracted increasing attention, aiming to enable robots to efficiently search multiple objects in unseen environments without training. For this task, the key lies in improving navigation policies from earlier searches to locate subsequent goals more efficiently. Recent methods automate this process through trajectory reflection and rule revision, enabling policy self-improvement. However, the mechanisms for diagnosing problems and generating revisions are typically designed by humans and remain fixed across iterations. Consequently, the system's capacity for continued self-improvement remains constrained, even as its navigation policy evolves. To this end, we propose RPSI-Nav, a Recursive Policy Self-Improvement framework for zero-shot multi-object navigation that revises both the navigation program and its revision mechanism. RPSI-Nav diagnoses search behavior, requests diagnostic views when existing observations cannot distinguish competing causes, and generates targeted navigation revisions. Feedback from subsequent goal searches is linked to the rules that produced these revisions, guiding changes to diagnosis, observation acquisition, and revision planning. The updated improver governs subsequent navigation improvements, forming a recursive loop in which RPSI-Nav revises its own improvement process. Across goals, the robot conditionally inherits accepted program updates and spatial memory, enabling search experience to improve both future navigation and adaptation. Benchmark experiments and qualitative real-robot evaluations validate RPSI-Nav's effectiveness.
BibTeX
@misc{li2026improve,
  author = {Yang Li and Ziju Han and Aming WU and Zihao Zhang and Sijia Zhang and Yahong Han and Qinghua Hu},
  title = {{Improve How You Improve: Recursive Policy Self-Improvement for Zero-Shot Open-Vocabulary Multi-Object Navigation}},
  year = {2026},
  eprint = {2611.23774},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23774},
}
Figure