ADAPT: Audio-Depth Action Planning with Transition-Aware Rewards for Lights-Out Industrial Navigation

Wenjun Huang, Shenghao Fu, Yian Jin, Yang Ni, Yifan Li, Lichi Li, Xingjian Diao, Pengfei Zhang, Yutong Song, Hanning Chen, Jiarui Cui, Tianyu Yang, Shilong Li, Yu Kong, Mohsen ImaniarXiv 2026 · arXiv

As industrial automation accelerates, we are seeing the rise of "lights-out" manufacturing—factories that operate with minimal human presence and, consequently, no need for visible illumination. While this saves energy, it creates a massive headache for modern robots. Most state-of-the-art navigation models are "vision-first," relying heavily on RGB cameras. When the lights go out, these robots effectively go blind.

A new paper, "ADAPT: Audio-Depth Action Planning with Transition-Aware Rewards for Lights-Out Industrial Navigation," proposes a solution: stop trying to see in the dark and start listening and feeling instead.

TL;DR

The authors introduce ADAPT, a framework that enables robots to navigate dark industrial environments using only spatial audio (to find the target) and depth sensing (to avoid obstacles). By introducing a novel Transition-Aware Reward Shaping (TARS) for reinforcement learning, they achieved a 90.6% success rate in unseen environments, proving that "RGB-free" navigation is a viable path for the future of automation.

What Problem Does This Paper Address?

The core challenge is illumination-independent navigation. Traditional Vision-Language-Action (VLA) models assume a well-lit world. In a dark factory, RGB data is just noise.

While depth sensors (like LiDAR or infrared) work in the dark, they only provide local geometric information—they can tell a robot there’s a crate in front of it, but not where the final destination is. Spatial audio provides the missing piece: a global cue. By "listening" to an acoustic beacon at the target, a robot can determine the general direction it needs to head. However, combining these two modalities into a reliable, closed-loop navigation policy is non-trivial, especially in noisy industrial settings.

Figure 1: Motivation of ADAPT. Existing RGB-based VLA agents depend on visible illumination and there- fore require continuous lighting in lights-out factories. ADAPT instead combines spatial audio for target di- rection with depth sensing for local obstacle geometry, enabling reliable and energy-efficient navigation with- out visible light.

Methods

The researchers developed a three-stage training pipeline to transform a Large Multimodal Model (LMM) into a specialized navigator:

  1. Spatial Audio Adaptation (SAA): Most LMMs are trained on mono audio. The authors added a lightweight adapter to help the model interpret 4-channel "First-Order Ambisonics" (FOA). This allows the model to estimate the "Direction of Arrival" (DoA) of a sound.
  2. Multimodal Supervised Fine-Tuning (SFT): The model is trained on expert demonstrations to jointly reason over audio (where to go) and depth (how to move safely).
  3. RL with TARS: This is the "secret sauce." Standard Reinforcement Learning (RL) rewards often use "geodesic distance" (the shortest path on a map). The problem? In a discrete action space, rotating in place doesn't change your position, so the robot gets zero reward for a "good" turn. The authors introduced Transition-Aware Reward Shaping (TARS), which calculates rewards based on the minimum number of actions (including turns) required to reach the goal. This provides much clearer feedback for policy optimization.
Figure 2: Overview of ADAPT. The model is trained in three stages: spatial-audio adaptation, multimodal SFT for audio-depth action prediction, and PPO-based closed-loop optimization with TARS. At inference time, the final policy directly maps spatial audio and depth observations to discrete navigation actions.

Datasets

The authors introduced DarkNav-Bench, a new benchmark built on the IndustryNav simulator. It features:

  • 24 Industrial Scenes: 10 for training, 14 kept entirely unseen for testing generalization.
  • Four Acoustic Difficulty Levels:
    • Clean: Direct path sound only.
  • Propagation: Includes reflections and occlusions from walls and machinery.
  • Diffuse Noise: Adds a constant background hum.
  • Localized Noise: The hardest setting, featuring distracting sounds from other moving robots, conveyors, and forklifts.

Key Insights and Ideas

  • Complementary Modalities: Audio acts as a "global compass," while depth acts as "local eyes." Neither is sufficient alone in the dark, but together they are powerful.
  • The Rotation Problem: In discrete navigation, turning is just as important as moving forward. By using a position-heading graph for rewards (TARS), the authors solved the "ambiguous credit assignment" problem that plagues standard geodesic rewards.
  • LMMs are Adaptable: You don't need to build a robot brain from scratch. By adding small adapters to existing Large Multimodal Models, you can "teach" them entirely new sensory capabilities like spatial hearing.

Key Results

ADAPT significantly outperformed existing baselines (like MAGNet and SAVi) and zero-shot LMMs:

  • Unseen "Clean" Scenes: Achieved a 90.6% success rate.
  • Localized Noise (Hardest): Maintained a 60.0% success rate, whereas traditional baselines often saw their performance collapse toward zero in noisy environments.
  • Efficiency: Thanks to TARS, the robot took more direct paths and made fewer unnecessary rotations compared to models trained with standard rewards.

Limitations and Future Work

While impressive, the work has some hurdles to clear before hitting the factory floor:

  • Sim-to-Real Gap: The study was conducted entirely in high-fidelity simulation. Real-world microphones have noise, calibration drift, and channel mismatches that weren't fully modeled.
  • Controlled Beacons: The target used a specific "chirp" signal. Real environments might have intermittent targets or distractors that sound very similar to the goal.
  • Privileged Information: The TARS reward requires a ground-truth map during training. Future work could explore "map-free" reward estimation.

The authors conclude that spatial audio and depth are not just "backups" for vision—they are primary, reliable modalities for the next generation of energy-efficient, autonomous industrial robots.

Summary updated

Abstract

Vision-language-action models for embodied navigation typically rely on RGB observations, limiting their applicability in lights-out industrial environments where visible illumination may be unavailable. We study illumination-independent navigation using spatial audio for target localization and depth for obstacle-aware geometry. To support this setting, we introduce DarkNav-Bench, a benchmark with 24 industrial scenes and four progressively challenging acoustic conditions. We further propose ADAPT, an audio-depth action-planning framework equipped with spatial-audio perception and optimized for closed-loop navigation through reinforcement learning (RL). Our key RL contribution, transition-aware reward shaping (TARS), defines a pose-aware shortest-action potential over a position-heading graph. Unlike conventional geodesic-progress rewards, TARS accounts for both feasible movements and in-place rotations, providing action-consistent feedback for policy optimization. Experiments show that ADAPT consistently outperforms training-based navigation baselines and zero-shot multimodal models on both seen and unseen scenes. On unseen environments, it achieves 90.6% success under Clean conditions and retains 60.0% success under the most challenging Localized Noise setting. These results demonstrate the potential of spatial audio and depth as complementary modalities for reliable RGB-free navigation in lights-out industrial environments.
BibTeX
@misc{huang2026adapt,
  author = {Wenjun Huang and Shenghao Fu and Yian Jin and Yang Ni and Yifan Li and Lichi Li and Xingjian Diao and Pengfei Zhang and Yutong Song and Hanning Chen and Jiarui Cui and Tianyu Yang and Shilong Li and Yu Kong and Mohsen Imani},
  title = {{ADAPT: Audio-Depth Action Planning with Transition-Aware Rewards for Lights-Out Industrial Navigation}},
  year = {2026},
  eprint = {2610.15956},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.15956},
}
Figure