As industrial automation accelerates, we are seeing the rise of "lights-out" manufacturing—factories that operate with minimal human presence and, consequently, no need for visible illumination. While this saves energy, it creates a massive headache for modern robots. Most state-of-the-art navigation models are "vision-first," relying heavily on RGB cameras. When the lights go out, these robots effectively go blind.
A new paper, "ADAPT: Audio-Depth Action Planning with Transition-Aware Rewards for Lights-Out Industrial Navigation," proposes a solution: stop trying to see in the dark and start listening and feeling instead.
TL;DR
The authors introduce ADAPT, a framework that enables robots to navigate dark industrial environments using only spatial audio (to find the target) and depth sensing (to avoid obstacles). By introducing a novel Transition-Aware Reward Shaping (TARS) for reinforcement learning, they achieved a 90.6% success rate in unseen environments, proving that "RGB-free" navigation is a viable path for the future of automation.
What Problem Does This Paper Address?
The core challenge is illumination-independent navigation. Traditional Vision-Language-Action (VLA) models assume a well-lit world. In a dark factory, RGB data is just noise.
While depth sensors (like LiDAR or infrared) work in the dark, they only provide local geometric information—they can tell a robot there’s a crate in front of it, but not where the final destination is. Spatial audio provides the missing piece: a global cue. By "listening" to an acoustic beacon at the target, a robot can determine the general direction it needs to head. However, combining these two modalities into a reliable, closed-loop navigation policy is non-trivial, especially in noisy industrial settings.
Methods
The researchers developed a three-stage training pipeline to transform a Large Multimodal Model (LMM) into a specialized navigator:
- Spatial Audio Adaptation (SAA): Most LMMs are trained on mono audio. The authors added a lightweight adapter to help the model interpret 4-channel "First-Order Ambisonics" (FOA). This allows the model to estimate the "Direction of Arrival" (DoA) of a sound.
- Multimodal Supervised Fine-Tuning (SFT): The model is trained on expert demonstrations to jointly reason over audio (where to go) and depth (how to move safely).
- RL with TARS: This is the "secret sauce." Standard Reinforcement Learning (RL) rewards often use "geodesic distance" (the shortest path on a map). The problem? In a discrete action space, rotating in place doesn't change your position, so the robot gets zero reward for a "good" turn. The authors introduced Transition-Aware Reward Shaping (TARS), which calculates rewards based on the minimum number of actions (including turns) required to reach the goal. This provides much clearer feedback for policy optimization.
Datasets
The authors introduced DarkNav-Bench, a new benchmark built on the IndustryNav simulator. It features:
- 24 Industrial Scenes: 10 for training, 14 kept entirely unseen for testing generalization.
- Four Acoustic Difficulty Levels:
- Clean: Direct path sound only.
- Propagation: Includes reflections and occlusions from walls and machinery.
- Diffuse Noise: Adds a constant background hum.
- Localized Noise: The hardest setting, featuring distracting sounds from other moving robots, conveyors, and forklifts.
Key Insights and Ideas
- Complementary Modalities: Audio acts as a "global compass," while depth acts as "local eyes." Neither is sufficient alone in the dark, but together they are powerful.
- The Rotation Problem: In discrete navigation, turning is just as important as moving forward. By using a position-heading graph for rewards (TARS), the authors solved the "ambiguous credit assignment" problem that plagues standard geodesic rewards.
- LMMs are Adaptable: You don't need to build a robot brain from scratch. By adding small adapters to existing Large Multimodal Models, you can "teach" them entirely new sensory capabilities like spatial hearing.
Key Results
ADAPT significantly outperformed existing baselines (like MAGNet and SAVi) and zero-shot LMMs:
- Unseen "Clean" Scenes: Achieved a 90.6% success rate.
- Localized Noise (Hardest): Maintained a 60.0% success rate, whereas traditional baselines often saw their performance collapse toward zero in noisy environments.
- Efficiency: Thanks to TARS, the robot took more direct paths and made fewer unnecessary rotations compared to models trained with standard rewards.
Limitations and Future Work
While impressive, the work has some hurdles to clear before hitting the factory floor:
- Sim-to-Real Gap: The study was conducted entirely in high-fidelity simulation. Real-world microphones have noise, calibration drift, and channel mismatches that weren't fully modeled.
- Controlled Beacons: The target used a specific "chirp" signal. Real environments might have intermittent targets or distractors that sound very similar to the goal.
- Privileged Information: The TARS reward requires a ground-truth map during training. Future work could explore "map-free" reward estimation.
The authors conclude that spatial audio and depth are not just "backups" for vision—they are primary, reliable modalities for the next generation of energy-efficient, autonomous industrial robots.
Summary updated