AcousticBEV: Acoustic BEV Object Localization for Intelligent Vehicles

Ziang Liu, Shiming Wang, Joseph Adeola, Julian F. P. KooijarXiv 2026 · arXiv

TL;DR

Researchers have developed AcousticBEV, a system that allows autonomous vehicles to "hear" and localize surrounding traffic in a Bird’s-Eye View (BEV) map using only a roof-mounted microphone array. By introducing the new AcousticScenes dataset, they demonstrate that acoustic sensing can effectively complement cameras and LiDAR, especially in challenging conditions like nighttime or fog.

What Problem Does This Paper Address?

Modern autonomous driving relies heavily on "vision" (cameras) and "active sensing" (LiDAR and Radar). However, these sensors have Achilles' heels: cameras struggle in low light or glare, and LiDAR performance drops in heavy fog or rain.

Humans, by contrast, often hear a car approaching a blind corner before they see it. While microphone arrays have been used for indoor robotics (like localizing a speaker), using them for high-speed, outdoor vehicle localization is incredibly difficult. The "ego-noise" of the driving vehicle (wind, tire friction) and the complex physics of sound propagation in open environments make it hard to pinpoint a moving sound source's distance and direction accurately. This paper tackles the challenge of Acoustic BEV Object Localization: estimating the metric 2D coordinates of nearby vehicles using sound alone.

Methods

The authors propose a holistic approach spanning hardware design and deep learning:

  1. Surround Microphone Array (SMA): They designed a custom, two-layer horizontal array of 32 MEMS microphones mounted on a vehicle's roof. This configuration provides 360° coverage and deliberate height/radial offsets to improve spatial resolution.
  2. AcousticBEV Architecture:
    • Feature Extraction: Raw audio is converted into Time-Frequency representations via Short-Time Fourier Transform (STFT).
  3. Ego-Velocity Conditioning: Recognizing that a vehicle's own speed changes the acoustic profile (Doppler shifts and wind noise), they use a FiLM (Feature-wise Linear Modulation) layer to condition the network on the ego-vehicle's velocity.
  4. Polar Heatmap Prediction: Instead of regressing coordinates directly, the model predicts a dense heatmap in polar coordinates (angle and distance).
  5. Uncertainty-Aware Labeling: They use 2D Polar Gaussian kernels for ground-truth labels. Crucially, they model the uncertainty as range-dependent: as a vehicle gets further away, the angular uncertainty decreases (it's a smaller point) but the distance uncertainty increases (sound attenuates).

Datasets

The authors introduced AcousticScenes, a first-of-its-kind multimodal dataset for this task:

  • Scale: 4,980 samples (2,117 static ego-vehicle, 2,863 dynamic).
  • Sensors: 32-channel audio synchronized with 128-beam LiDAR and multi-view cameras.
  • Ground Truth: High-resolution 3D bounding boxes derived from LiDAR, providing precise metric supervision for the acoustic model.
  • Diversity: Data was collected at various ego-speeds (0, 10, 30, and 50 km/h) in real-world driving scenarios in Delft, Netherlands.

Key Insights and Ideas

  • Ego-Speed Matters: The inclusion of the FiLM velocity encoder was vital. Without it, the model struggled to distinguish between the sound of a nearby car and the increased wind/tire noise at higher speeds.
  • Polar is Natural for Sound: Localizing sound is inherently about Direction-of-Arrival (DOA) and distance. By formulating the task in polar space rather than Cartesian (X, Y) space, the model architecture better aligns with the physical nature of acoustic signals.
  • Two Layers are Better Than One: The ablation study showed that a two-layer microphone array significantly outperformed a single-layer version, likely because the vertical offset helps resolve complex wave interference patterns.
  • Complementary Modality: Qualitative results show the system successfully localizing vehicles at night where cameras see only blackness or headlight glare.

Key Results

AcousticBEV was compared against classical signal processing (Beamforming, MUSIC) and existing learning-based methods (SELDNet):

  • Localization Performance: AcousticBEV achieved an F1 score of 47.7% on "Easy" scenarios (0-1 vehicles) and 45.1% on "Difficult" scenarios (2-3 vehicles).
  • Accuracy: For correctly detected vehicles, the Mean Average Distance Error (mADE) was approximately 1.76 to 1.81 meters, and the Mean Average Angle Error (mAAE) was roughly 1.8°.
  • Efficiency: The model is lightweight (3.3M parameters) and runs at over 500 FPS on a standard GPU, making it highly suitable for real-time automotive applications.

Limitations and Future Work

While a major step forward, the authors note several areas for growth:

  • Traffic Density: The current dataset lacks highly congested urban environments or complex junctions (like narrow T-junctions).
  • Sensor Fusion: This paper focuses on microphones as the sole sensor to prove their utility. The next logical step is late-fusion, where acoustic cues are integrated with LiDAR and Camera data to create a more robust, all-weather perception stack.
  • Environmental Factors: Future work could explore how different road surfaces or extreme weather (heavy rain) impact the "acoustic visibility" of the road.

Summary updated

Abstract

Modern autonomous driving and driver assistance systems rely predominantly on cameras, radars and LiDARs for environmental perception. However, these modalities face different environmental limitations, including poor illumination for cameras and adverse weather for LiDAR. Acoustic sensing, particularly using microphone arrays, could provide complementary perception capabilities for autonomous driving, but has remained largely underexplored. In this work, we design a horizontally arranged microphone array that functions as the "ears" of a vehicle, enabling localization of nearby moving vehicles in a bird's-eye view (BEV) representation that captures both their relative direction and distance. Using this array, we collect a real-world driving dataset, AcousticScenes, which contains 2,117 static and 2,863 dynamic samples with synchronized spatial acoustic signals, LiDAR point clouds and 3D bounding box annotations. Building on this dataset, we propose AcousticBEV, a learning-based method for acoustic BEV object localization, which estimates the center locations of moving traffic participants in BEV from the microphone-array signals, conditioned on the ego vehicle's own speed. Experimental results demonstrate that AcousticBEV outperforms both classical signal-processing baselines and other learning-based methods, achieving F1 scores of $47.7 \pm 1.2$% on "Easy" scenarios and $45.1 \pm 1.4$% on "Difficult" cases. Our findings demonstrate that acoustic signals from a low-cost, vehicle-mounted microphone array can localize nearby moving vehicles in both azimuth and distance.
BibTeX
@misc{liu2026acousticbev,
  author = {Ziang Liu and Shiming Wang and Joseph Adeola and Julian F. P. Kooij},
  title = {{AcousticBEV: Acoustic BEV Object Localization for Intelligent Vehicles}},
  year = {2026},
  eprint = {2611.19381},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19381},
}
Figure