R2H: Building Multimodal Navigation Helpers that Respond to Help

Yue Fan, Kaizhi Zheng, Jing Gu, Xin Eric Wang

Introduction

In the field of Vision-and-Language Navigation (VLN), research has traditionally focused on the "performer"—the agent tasked with executing instructions to reach a destination. However, a critical yet under-explored counterpart is the "helper": an intelligent agent capable of observing the environment, accessing global information (like maps or goal locations), and providing conversational guidance. Such helpers are essential for real-world applications, particularly as accessibility tools for individuals with visual impairments or for navigating unfamiliar complex environments.

Developing these helper agents presents two primary challenges. First, evaluation is difficult; traditionally, assessing a helper's effectiveness required a human-in-the-loop to follow the instructions, which is neither scalable nor cost-effective. Second, modeling human-like assistance is complex because natural dialogue is often redundant and noisy, making it hard for models to extract and provide concise, actionable directions.

The R2H Benchmark

To address these challenges, we introduce the Respond to Help Requests (R2H) benchmark. R2H provides a framework for evaluating multimodal navigation helpers by pairing them with a fixed, pre-trained performer agent. This setup allows for automated, reproducible evaluation where the helper's quality is directly measured by the performer's success in reaching the goal.

The benchmark consists of two distinct tasks:

  1. Respond to Dialog History (RDH): This task evaluates the helper's ability to generate an informative response based on a static history of previous dialogue and visual observations.
  2. Respond during Interaction (RdI): This task assesses the helper's effectiveness in a dynamic, multi-turn setting, where it must consistently cooperate with the performer to complete a navigation task.

SeeRee: A Multimodal Helper Agent

Alongside the benchmark, we propose SeeRee (See and Respond), a novel multimodal transformer-based agent designed specifically for the helper role. SeeRee incorporates two key technical innovations:

Conditional Optimized Sparse (COS) Attention

Navigation tasks involve processing long sequences of visual frames, many of which are redundant. SeeRee utilizes a Conditional Optimized Sparse (COS) Attention Mask, a learnable, non-binary mechanism that identifies and focuses on critical visual details. By ignoring irrelevant frames, the model improves both its computational efficiency and its ability to ground language in the most pertinent environmental features.

Parse by Step Training

Human dialogue in navigation datasets is often conversational but structurally "noisy." To improve instruction quality, we implemented a Parse by Step strategy. Using GPT-3, we preprocessed raw human responses into structured, step-by-step instructions. Training SeeRee on this cleaned, task-oriented data significantly enhanced its ability to provide clear and useful guidance compared to models trained on raw dialogue.

Experimental Setup and Datasets

The R2H benchmark harmonizes three major embodied AI datasets into a unified format:

  • CVDN: Indoor navigation within photo-realistic environments.
  • DialFRED: Navigation and object interaction in synthetic environments.
  • AVDN: Aerial drone navigation featuring a continuous control space.

This diversity ensures that helper agents are tested across various visual domains and interaction complexities.

Key Insights and Results

Our evaluation of SeeRee against existing baselines (such as RMM) and zero-shot Multimodal Large Language Models (such as mPLUG-Owl) yielded several critical insights:

  • Utility Over Mimicry: A significant finding was that linguistic similarity metrics (like BLEU or ROUGE) do not correlate strongly with task success. A helper that sounds "human" is not necessarily a good guide. The effectiveness of a helper is defined by the utility of the information provided, not the phrasing.
  • Superior Performance: SeeRee consistently outperformed all baselines across the three datasets. On the CVDN dataset, SeeRee achieved a Goal Progress (GP) of 6.5, approaching the human annotator benchmark of 6.9.
  • Human Evaluation: In real-time trials where humans acted as performers, SeeRee achieved the highest task completion rate (9.89 GP), surpassing both the MLLM (8.44 GP) and the baseline (6.32 GP). While humans noted that MLLMs sounded more "natural," they rated SeeRee as more "faithful" and accurate in its guidance.

Limitations and Future Directions

While R2H and SeeRee represent a step forward, several limitations remain. The helper's performance is currently coupled with the capabilities of the fixed performer agent; errors by the performer can unfairly penalize the helper. Additionally, there is a persistent "sim-to-real" gap, as current simulators do not fully capture the complexity of dynamic real-world environments.

Future research will focus on developing more robust performer agents and exploring the deployment of SeeRee-like models in physical robotic systems to assist humans in real-time, real-world navigation.

Summary updated

Abstract

Intelligent navigation-helper agents are critical as they can navigate users in unknown areas through environmental awareness and conversational ability, serving as potential accessibility tools for individuals with disabilities. In this work, we first introduce a novel benchmark, Respond to Help Requests (R2H), to promote the development of multi-modal navigation helpers capable of responding to requests for help, utilizing existing dialog-based embodied datasets. R2H mainly includes two tasks: (1) Respond to Dialog History (RDH), which assesses the helper agent's ability to generate informative responses based on a given dialog history, and (2) Respond during Interaction (RdI), which evaluates the effectiveness and efficiency of the response during consistent cooperation with a task performer. Furthermore, we explore two approaches to construct the navigation-helper agent, including fine-tuning a novel task-oriented multi-modal response generation model that can see and respond, named SeeRee, and employing a multi-modal large language model in a zero-shot manner. Analysis of the task and method was conducted based on both automatic benchmarking and human evaluations. Project website: https://sites.google.com/view/response2helprequests/home.
BibTeX
@inproceedings{fan2023r,
  author = {Yue Fan and Kaizhi Zheng and Jing Gu and Xin Eric Wang},
  title = {{R2H: Building Multimodal Navigation Helpers that Respond to Help}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.915},
}
Figure