AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments

Sudipta Paul, Amit Roy-Chowdhury, Anoop Cherian

TL;DR

AVLEN is a multimodal embodied navigation agent that combines audio-visual cues with the ability to interactively query a human "oracle" for help. Using a hierarchical reinforcement learning framework, the agent learns to autonomously navigate 3D environments and strategically ask for natural language instructions when it encounters uncertainty, significantly improving success rates in complex, noisy scenarios.

What Problem Does This Paper Address?

Imagine a robot tasked with finding a "thud" in a large house. In traditional AI research, this problem is usually split into two silos: Vision-and-Language Navigation (VLN), where agents follow a set of instructions but are "deaf," and Audio-Visual Navigation (AVN), where agents hunt for a sound source but cannot understand language.

Top-down view of the AVLEN agent navigating a 3D environment, querying an oracle for help, and reaching the sound source.

In the real world, sounds are often sporadic (a door slams once and then it is quiet) or ambiguous (multiple appliances running at once). Current agents often get "lost" in these scenarios. While some previous works allowed agents to ask for help, they relied on rigid heuristics—like asking for help only when a probability score dropped below a certain threshold. The authors of AVLEN argue that an agent should learn the value of a human's advice and decide for itself when the cost of a query is worth the navigation boost.

Methods

The core of the paper is a Multimodal Hierarchical Reinforcement Learning framework. Instead of one giant neural network, AVLEN uses a two-level hierarchy to manage its navigation and querying behavior:

Architecture diagram of AVLEN's Multimodal Hierarchical Reinforcement Learning framework.
  1. High-Level Policy ($\pi_q$): This acts as the "manager." At each step, it looks at the agent’s history and current audio-visual state to decide: "Should I keep navigating on my own, or should I ask the oracle for help?" It is trained with a reward function that balances the desire to reach the goal with a penalty for "bothering" the oracle too much.
  2. Low-Level Policies:
    • Audio-Visual Policy ($\pi_g$): Navigates based on RGB-D images and binaural (stereo) audio. It uses a Transformer-based architecture to remember past observations.
  3. Language Policy ($\pi_l$): If the manager decides to ask for help, the oracle provides a short instruction (e.g., "Go towards the dining table"). This policy uses CLIP embeddings and a Transformer to translate that text into a sequence of physical actions.

To train this system without a human sitting at a computer for millions of iterations, the authors used a "Speaker" model—a separate AI trained to generate human-like instructions based on the shortest path to the goal.

Datasets

The researchers evaluated AVLEN using the SoundSpaces framework, which is built on top of Matterport3D.

  • Matterport3D: Provides 85 highly detailed, real-world 3D scans of houses.
  • SoundSpaces: Simulates how sound actually travels through those specific rooms, accounting for reverberation and occlusion by walls.
  • SAVi (Semantic Audio-Visual Navigation): A benchmark where the agent must find an object (like a cat or a telephone) that makes a sound sporadically.

Key Insights and Ideas

  • Learning Uncertainty: Rather than using a hard-coded threshold to trigger a help request, AVLEN treats "asking for help" as just another action in a Reinforcement Learning setting. This allows the agent to implicitly learn the "uncertainty" of its own audio-visual sensors.
  • Judicial Querying: The agent doesn't just spam the oracle at the start. The results show it learns to "save" its queries for when the sound stops or when it enters a confusing part of the house.
  • Language as a Shortcut: Natural language provides high-level semantic information that raw audio cannot. A single instruction like "turn left at the sofa" can resolve ambiguity that might take an audio-only agent minutes of wandering to figure out.

Key Results

The authors tested AVLEN in three tough scenarios: sporadic sounds, unheard sounds (objects not seen during training), and distractor sounds (background noise).

  • Performance Boost: In the "unheard sound" setting, AVLEN achieved a 26.2% success rate, outperforming the state-of-the-art SAVi baseline (24.8%).
  • The "Oracle" Gap: When the agent was given ground-truth actions instead of language instructions, the success rate jumped to 36.7%. This suggests that the agent is very good at knowing when to ask, but still has room to improve at understanding the language it receives.
  • Robustness to Noise: In environments with distractor sounds, AVLEN's ability to ask for help became even more valuable, showing a more significant lead over non-interactive baselines.

Limitations and Future Work

  • Language Understanding: As noted by the "Oracle Gap," the current bottleneck is the language-to-action mapping. Future work could involve more sophisticated pre-trained language models.
  • Oracle Realism: The simulated oracle always provides perfect, shortest-path instructions. Real humans might be vague, slow to respond, or even occasionally wrong.
  • Dataset Diversity: The Matterport3D dataset is primarily composed of North American homes. The agent's "common sense" about where a dining table is located might not translate to different cultures or architectural styles.
  • Language Constraint: The model is currently limited to English instructions.

Summary updated

Abstract

Recent years have seen embodied visual navigation advance in two distinct directions: (i) in equipping the AI agent to follow natural language instructions, and (ii) in making the navigable world multimodal, e.g., audio-visual navigation. However, the real world is not only multimodal, but also often complex, and thus in spite of these advances, agents still need to understand the uncertainty in their actions and seek instructions to navigate. To this end, we present AVLEN -- an interactive agent for Audio-Visual-Language Embodied Navigation. Similar to audio-visual navigation tasks, the goal of our embodied agent is to localize an audio event via navigating the 3D visual world; however, the agent may also seek help from a human (oracle), where the assistance is provided in free-form natural language. To realize these abilities, AVLEN uses a multimodal hierarchical reinforcement learning backbone that learns: (a) high-level policies to choose either audio-cues for navigation or to query the oracle, and (b) lower-level policies to select navigation actions based on its audio-visual and language inputs. The policies are trained via rewarding for the success on the navigation task while minimizing the number of queries to the oracle. To empirically evaluate AVLEN, we present experiments on the SoundSpaces framework for semantic audio-visual navigation tasks. Our results show that equipping the agent to ask for help leads to a clear improvement in performances, especially in challenging cases, e.g., when the sound is unheard during training or in the presence of distractor sounds.
BibTeX
@inproceedings{paul2022avlen,
  author = {Sudipta Paul and Amit Roy-Chowdhury and Anoop Cherian},
  title = {{AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=A6r7jiTFy6},
}
Figure