AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments

Sudipta Paul, Amit K. Roy-Chowdhury, Anoop Cherian

TL;DR

This paper introduces AVLEN, an embodied AI agent that navigates 3D environments using audio, visual, and natural language cues. AVLEN can intelligently decide when to ask a human "oracle" for help via natural language instructions, significantly improving its navigation performance, especially in challenging scenarios with unheard or distracting sounds.

What Problem Does This Paper Address?

Current embodied AI agents for navigation often fall into two camps: those that follow natural language instructions (vision-and-language navigation, VLN) and those that use audio-visual cues to find sound sources. However, the real world is a complex interplay of all these modalities. Agents often face uncertainty, making it difficult to complete tasks solely based on pre-programmed senses. This paper addresses the challenge of creating an AI agent that can not only navigate using a combination of audio and visual information but also intelligently seek human assistance through natural language when it encounters ambiguity or difficulty. This is crucial for building more robust and human-collaborative robotic systems.

Methods

AVLEN employs a multimodal hierarchical reinforcement learning (RL) framework. This means the agent has a layered decision-making process:

  • High-Level Policy ($\pi_a$): This policy acts as the "brain" deciding whether to rely on audio-visual cues for navigation or to query the oracle for help. It learns to balance the need for assistance with maintaining agent autonomy.
  • Low-Level Policies:
    • Audio-Visual Navigation Policy ($\pi_g$): This policy uses the agent's current audio and visual observations to select navigation actions, aiming to reach the sound source directly.
  • Language-Based Navigation Policy ($\pi_l$): When the agent queries the oracle, this policy takes the natural language instruction and translates it into a sequence of navigation actions.

The system is trained end-to-end using RL, with rewards designed to encourage successful navigation while penalizing excessive queries to the oracle. The agent's state is represented by a memory module that stores a history of observations, including visual (RGB and depth), audio, previous actions, and pose.

Datasets

The experiments are conducted using the SoundSpaces platform, which simulates realistic 3D environments with audio events. The specific dataset used is the semantic audio-visual navigation dataset from Chen et al. [5], which includes sounds from 21 semantic object categories within Matterport3D scans. For training the language policy ($\pi_l$), the authors repurpose trajectory-instruction pairs from the VLN dataset [2].

Key Insights and Ideas

  • Intelligent Querying is Key: The most significant insight is that simply having access to natural language instructions isn't enough. The agent must learn when to ask for help. AVLEN's hierarchical policy learns to query judiciously, leading to substantial performance gains, especially in difficult scenarios.
  • Bridging Modalities: AVLEN demonstrates a powerful synergy between audio, visual, and language modalities. It's not just about using them in parallel but about intelligently switching between them based on the agent's confidence and the task's demands.
  • Free-Form Language Instructions: The ability to handle abstract, free-form natural language instructions from an oracle is a critical step towards more natural human-robot interaction.
  • Hierarchical RL for Complex Tasks: The hierarchical approach effectively decomposes the complex navigation problem into manageable sub-problems (deciding to query vs. navigating, and then executing navigation based on the chosen modality).

Key Results

  • AVLEN achieves a +2.2% and +1.6% absolute gain in success rate and success-when-silent (SWS) respectively, compared to the best baseline (SAVi [5]) in heard sound settings.
  • In unheard sound settings, AVLEN shows a +1.4% and +1.1% absolute gain in success rate and SWS over SAVi.
  • When provided with ground truth navigation actions (instead of language) as oracle feedback, AVLEN shows a dramatic improvement of nearly +15% for heard sounds and +12% for unheard sounds, highlighting the potential of the language policy ($\pi_l$).
  • AVLEN outperforms other query-triggering baselines (Uniform, Random, Model Uncertainty) across various metrics, demonstrating the effectiveness of its learned querying strategy.
  • The agent shows robustness to distractor sounds, achieving higher cumulative success when sounds are silent for longer periods.

Limitations and Future Work

  • Language Policy Dependency: The performance of AVLEN is significantly reliant on the quality of its language-based navigation policy ($\pi_l$) and the speaker model used to generate instructions. If these are imperfect (e.g., irrelevant or ambiguous instructions), performance can suffer.
  • Dataset Bias: The current work uses datasets that primarily represent North American houses, potentially leading to a model bias.
  • Societal Concerns: While AVLEN enables more natural human-robot interaction, the paper notes a gap in how the agent interprets and translates language into actions, which could be a point of societal concern.

Future work could focus on improving the language policy, exploring more diverse environments, and addressing the potential societal implications of language interpretation in AI agents.

Summary updated

Abstract

Recent years have seen embodied visual navigation advance in two distinct directions: (i) in equipping the AI agent to follow natural language instructions, and (ii) in making the navigable world multimodal, e.g., audio-visual navigation. However, the real world is not only multimodal, but also often complex, and thus in spite of these advances, agents still need to understand the uncertainty in their actions and seek instructions to navigate. To this end, we present AVLEN~ -- an interactive agent for Audio-Visual-Language Embodied Navigation. Similar to audio-visual navigation tasks, the goal of our embodied agent is to localize an audio event via navigating the 3D visual world; however, the agent may also seek help from a human (oracle), where the assistance is provided in free-form natural language. To realize these abilities, AVLEN uses a multimodal hierarchical reinforcement learning backbone that learns: (a) high-level policies to choose either audio-cues for navigation or to query the oracle, and (b) lower-level policies to select navigation actions based on its audio-visual and language inputs. The policies are trained via rewarding for the success on the navigation task while minimizing the number of queries to the oracle. To empirically evaluate AVLEN, we present experiments on the SoundSpaces framework for semantic audio-visual navigation tasks. Our results show that equipping the agent to ask for help leads to a clear improvement in performance, especially in challenging cases, e.g., when the sound is unheard during training or in the presence of distractor sounds.
BibTeX
@inproceedings{paul2022avlen,
  author = {Sudipta Paul and Amit K. Roy-Chowdhury and Anoop Cherian},
  title = {{AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=A6r7jiTFy6},
}
Figure