Bat-G net: Bat-inspired High-Resolution 3D Image Reconstruction using Ultrasonic Echoes

Gunpil Hwang, Seohyeon Kim, Hyeon-Min BaeNeurIPS 2019 · OpenReview

Introduction

Researchers have developed Bat-G net, a deep learning architecture that mimics a bat’s auditory system to reconstruct high-resolution 3D images from ultrasonic echoes. By using hyperbolic frequency-modulated (HFM) chirps and a biomimetic neural network, the system can "see" objects in 3D even in environments where traditional cameras and LiDAR fail, such as dense fog or total darkness.

The Challenge of Adverse Environments

Autonomous systems like drones and self-driving cars currently rely on a suite of sensors: RGB-D cameras and LiDAR provide high-resolution data but are easily blinded by bad weather or poor lighting. Conversely, RADAR and standard ultrasonic sensors are robust to environmental conditions but typically only provide low-resolution ranging information (distance, not shape).

There is a significant gap for a sensor that is both high-resolution and environmentally robust. The authors look to nature for a solution: bats. Bats use sophisticated echolocation to navigate and hunt in complete darkness, processing complex echoes to perceive the 3D shape of their surroundings.

System overview demonstrating the robustness of ultrasound in fog and darkness compared to RGB-D cameras, alongside the Bat-G network architecture.

The challenge is translating this biological "inverse problem"—turning sound waves back into spatial geometry—into a reliable computational model.

Biomimetic Methods and Architecture

The authors developed a full-stack solution consisting of a custom sensor and a specialized neural network:

  1. Bat-I Sensor: A hardware module featuring one ultrasonic electrostatic speaker (the "mouth") and four microphones (the "ears"). It broadcasts Hyperbolic FM (HFM) chirps ranging from 20 to 120 kHz. HFM chirps are used because they are Doppler-tolerant and allow for high spatial resolution through pulse compression.
  2. Bat-G Network: A feed-forward neural network designed to emulate the bat's central auditory pathway.
  3. The Encoder: Mimics the brainstem and midbrain. It uses deformable convolutions to extract temporal and spectral features from spectrograms. It processes signals through two distinct pathways (short and long window STFTs) to capture different "cues" in the echoes.
  4. The Decoder: Inspired by the expansive path of a U-Net, this component projects the low-dimensional features from the encoder into a volumetric $64 \times 64 \times 64$ voxel grid.
  5. Data Processing: To manage the high data rate, the system uses "ROI pooling," focusing only on the time window where an object is likely to be (based on a 1.48m distance), reducing input dimensions by 98%.

To train the network, the researchers created a custom dataset called ECHO-4CH. It contains 16.2k unique geometric configurations (cubes, cones, spheres, etc.) created using building blocks and 3D printing, resulting in 81k measured echo data points. The ground truth consists of 3D CAD models voxelized into $64^3$ grids.

Key Insights and Ideas

  • Bio-mimicry as Architecture: Instead of a generic CNN, the network's hierarchy directly maps to biological structures like the Cochlear Nucleus and Superior Olivary Complex. This suggests that biological processing structures are inherently well-suited for solving specific inverse problems like echolocation.
  • Spectral vs. Temporal Cues: The paper highlights that bats don't just listen for "pings." They analyze "notches" and "nulls" in the frequency spectrum (spectral cues) to resolve fine details that are too small for time-of-flight measurements (temporal cues) to catch.
Signal processing pipeline illustrating how the system extracts spectral and temporal cues from ultrasonic echoes.
  • Robustness to Occlusion: The system successfully reconstructed objects in conditions (3 lx light, >999 $\mu g/m^3$ fog) where a Microsoft Kinect (RGB-D) failed completely.

Key Results

The Bat-G net demonstrated impressive accuracy in 3D reconstruction, achieving a Precision of 0.896, Recall of 0.899, and an F1-score of 0.895. It significantly outperformed a standard Stacked Auto-Encoder (SAE) baseline, showing a 5.4% improvement in F1-score when using 4-channel data.

An ablation study further validated the biomimetic design: removing the "biomimetic" connections led to a 5.1% drop in performance. Furthermore, using only one microphone (1-channel) caused the F1-score to plummet by nearly 28%, proving that multi-channel "binaural" input is essential for accurate 3D perception.

Limitations and Future Work

While the results are promising, the authors noted two main hurdles. First, if an object's surface area is too small or angled away from the sensor, the reflective area seen by the "ears" is insufficient to retrieve full edge information. Second, multipath interference (where sound bounces off multiple surfaces before returning) can confuse the network, leading it to erroneously represent the shape of the object.

The current study focused on isolated geometric objects at a fixed distance. Future work will need to address more complex, cluttered scenes and dynamic distances to make this technology viable for real-world robotics.

Summary updated

Abstract

In this paper, a bat-inspired high-resolution ultrasound 3D imaging system is presented. Live bats demonstrate that the properly used ultrasound can be used to perceive 3D space. With this in mind, a neural network referred to as a Bat-G network is implemented to reconstruct the 3D representation of target objects from the hyperbolic FM (HFM) chirped ultrasonic echoes. The Bat-G network consists of an encoder emulating a bat's central auditory pathway, and a 3D graphical visualization decoder. For the acquisition of the ultrasound data, a custom-made Bat-I sensor module is used. The Bat-G network shows the uniform 3D reconstruction results and achieves precision, recall, and F1-score of 0.896, 0.899 and 0.895, respectively. The experimental results demonstrate the implementation feasibility of a high-resolution non-optical sound-based imaging system being used by live bats. The project web page (https://sites.google.com/view/batgnet) contains additional content summarizing our research.
BibTeX
@inproceedings{hwang2019bat,
  author = {Gunpil Hwang and Seohyeon Kim and Hyeon-Min Bae},
  title = {{Bat-G net: Bat-inspired High-Resolution 3D Image Reconstruction using Ultrasonic Echoes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=DkKHjcBn56},
}
Figure