Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Grounding

Minyoung Hwang, Jaeyeon Jeong, Minsoo Kim, Yoonseon Oh, Songhwai Oh

Abstract

Vision-and-Language Navigation (VLN) requires agents to execute complex natural language instructions within unfamiliar indoor environments. A significant challenge in current VLN systems is their brittleness; a single incorrect turn often leads to an irrecoverable path. This paper introduces Meta-Explore, a hierarchical navigation framework designed to enhance agent robustness through intelligent error recovery. By utilizing a novel Scene Object Spectrum (SOS) representation—which leverages Fourier Transforms to capture structural object layouts—the agent can effectively scout unvisited frontiers to realign with its goal, rather than relying on inefficient backtracking or unrealistic teleportation.

1. Introduction

The goal of Vision-and-Language Navigation is for an autonomous agent to navigate to a target location based on natural language directives, such as "Walk past the kitchen, turn right at the table, and wait by the window." Despite advancements in deep reinforcement learning and transformer-based architectures, most agents struggle when they deviate from the intended path.

Traditional error-handling methods typically involve "homing" (backtracking to a previous state) or "jumping" (teleporting to a known node). However, homing is often repetitive and inefficient, while jumping is physically impossible for real-world robotic platforms. Meta-Explore addresses this by enabling agents to recognize when they are lost and proactively explore their surroundings to find a corrective path based on semantic landmarks.

2. The Meta-Explore Framework

Meta-Explore employs a hierarchical architecture that manages navigation through three integrated components:

2.1 Mode Selector

The Mode Selector acts as a high-level planner that monitors the agent's progress relative to the instructions. If the system detects a significant deviation or lack of progress, it switches the agent's state from "Exploration" (standard instruction following) to "Exploitation" (corrective search).

2.2 Topological Mapping

As the agent moves, it constructs a topological graph of the environment. This map tracks visited nodes and identifies "frontiers"—unvisited but observable areas. This graph allows the agent to maintain a global spatial memory, which is crucial for making informed decisions when the local visual input is ambiguous.

2.3 Scene Object Spectrum (SOS)

The core innovation of Meta-Explore is the Scene Object Spectrum. Rather than processing raw pixel data, which can be noisy and difficult to generalize across different architectural styles, the agent generates binary masks for detected object categories (e.g., chairs, tables, appliances). It then applies a 2D Fast Fourier Transform (FFT) to these masks.

This spectral-domain approach offers two primary advantages:

  1. Rotation Invariance: The frequency domain captures the structural "signature" of a room's layout, making it easier to identify landmarks regardless of the agent's orientation.
  2. Structural Reasoning: SOS focuses on the spatial frequency and arrangement of objects, providing a robust signal for matching the current scene to the landmarks described in the navigation instructions.

3. Exploitation and Navigation Scoring

When in "Exploitation" mode, the agent evaluates unvisited nodes on its topological map using a Navigation Score. This score is derived from the SOS features, allowing the agent to identify which frontier node most likely aligns with the next steps of the language instruction. This allows the agent to "correct" its trajectory by moving toward promising new areas rather than simply re-treading old ground.

4. Experimental Evaluation

The researchers evaluated Meta-Explore across several diverse benchmarks, covering both discrete and continuous environments:

  • Discrete Environments (R2R, SOON, REVERIE): These benchmarks test fine-grained instruction following and goal-oriented navigation toward specific remote objects.
  • Continuous Environments (VLN-CE): Utilizing the Habitat/Gibson simulators, this task requires navigation in photo-realistic environments with noisy, continuous movement, closely mimicking real-world robotics.

4.1 Key Results

Meta-Explore demonstrated state-of-the-art performance across all tested benchmarks:

  • SOON Benchmark: Achieved a 17.1% improvement in success rate and a 20.6% increase in Success weighted by Path Length (SPL).
  • R2R Benchmark: Outperformed existing hierarchical baselines (such as SMNA and Regretful-Agent) by at least 16.4% in success rate.
  • Generalization: The method showed superior performance in "Unseen" environments, suggesting that SOS features help the agent understand the concept of room layouts rather than memorizing specific visual instances.
  • Continuous Robustness: In VLN-CE, Meta-Explore outperformed baselines by 19.5% in success rate, proving its efficacy in handling noisy, realistic movement.

5. Discussion and Limitations

While Meta-Explore represents a significant step forward in robust VLN, it remains dependent on the quality of the underlying object detector. If the detector fails to identify key landmarks, the SOS feature cannot be accurately constructed. Additionally, noise in dataset metadata (such as typos or irrelevant categories in the REVERIE dataset) can impact the object-based parsing.

6. Conclusion

Meta-Explore shifts the paradigm of VLN from "perfect execution" to "intelligent recovery." By treating self-correction as a primary function and utilizing the frequency domain for semantic grounding, the framework provides a more efficient and realistic path for robotic navigation in complex, real-world homes. Future work will focus on extending this approach to long-horizon tasks and integrating a broader range of semantic cues.

Summary updated

Abstract

The main challenge in vision-and-language navigation (VLN) is how to understand natural-language instructions in an unseen environment. The main limitation of conventional VLN algorithms is that if an action is mistaken, the agent fails to follow the instructions or explores unnecessary regions, leading the agent to an irrecoverable path. To tackle this problem, we propose Meta-Explore, a hierarchical navigation method deploying an exploitation policy to correct misled recent actions. We show that an exploitation policy, which moves the agent toward a well-chosen local goal among unvisited but observable states, outperforms a method which moves the agent to a previously visited state. We also highlight the demand for imagining regretful explorations with semantically meaningful clues. The key to our approach is understanding the object placements around the agent in spectral-domain. Specifically, we present a novel visual representation, called scene object spectrum (SOS), which performs category-wise 2D Fourier transform of detected objects. Combining exploitation policy and SOS features, the agent can correct its path by choosing a promising local goal. We evaluate our method in three VLN benchmarks: R2R, SOON, and REVERIE. Meta-Explore outperforms other baselines and shows significant generalization performance. In addition, local goal search using the proposed spectral-domain SOS features significantly improves the success rate by 17.1% and SPL by 20.6% for the SOON benchmark.
BibTeX
@inproceedings{hwang2023meta,
  author = {Minyoung Hwang and Jaeyeon Jeong and Minsoo Kim and Yoonseon Oh and Songhwai Oh},
  title = {{Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Grounding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure