BirdSoundsDenoising: Deep Visual Audio Denoising for Bird Sounds

Youshan Zhang, Jialu Li

Introduction

Audio denoising remains a fundamental challenge in signal processing, with applications ranging from bioacoustics and environmental monitoring to telecommunications. Traditionally, this task has been approached through spectral subtraction or regression-based deep learning models that attempt to reconstruct clean 1D waveforms. However, these methods often struggle with "natural" noise—complex, non-stationary sounds like wind, rain, or mechanical interference—and frequently produce artifacts or "mushy" audio quality.

In this work, the authors propose a paradigm shift: treating audio denoising as a 2D semantic segmentation problem. By transforming audio into spectrograms, the task of removing noise becomes a computer vision problem where the goal is to distinguish the visual "shape" of a bird song from the background noise.

The DVAD Framework

The proposed Deep Visual Audio Denoising (DVAD) framework leverages the power of state-of-the-art computer vision architectures to clean audio signals. The process follows a four-step pipeline:

  1. Time-Frequency Transformation: The 1D raw audio signal is converted into a 2D spectrogram using the Short-Time Fourier Transform (STFT). This "audio image" represents frequency over time.
  2. Semantic Segmentation: Instead of predicting pixel intensities (regression), the model performs semantic segmentation. It classifies each pixel in the spectrogram as either "signal" (bird sound) or "noise" (background). The authors evaluated several architectures, including DeepLabV3, U-Net, and SegNet.
  3. Binary Masking: The model generates a binary mask. By multiplying the original noisy spectrogram by this mask, the background noise is effectively "zeroed out" while the structural integrity of the bird song is preserved.
  4. Signal Reconstruction: The cleaned spectrogram is converted back into a 1D audio waveform using the Inverse Short-Time Fourier Transform (ISTFT).

The BirdSoundsDenoising Dataset

A significant contribution of this research is the creation of the BirdSoundsDenoising dataset. Most existing denoising models are trained on clean audio mixed with synthetic white noise, which fails to generalize to real-world environments.

The authors curated 15,300 recordings from xeno-canto.org, resulting in 14,120 high-quality samples after rigorous filtering. This dataset is unique because it features natural background noises such as waterfalls, heavy rain, and wind. To facilitate the massive labeling effort required for segmentation, the researchers employed a few-shot generalization strategy: they manually labeled a small subset (140 images) and used a preliminary model combined with a custom refinement tool, ImageMask, to generate high-fidelity labels for the remaining 14,000+ images.

Key Insights

  • Visual Distinctness: In the frequency domain, bird songs exhibit distinct geometric patterns—chirps appear as sharp vertical or curved lines, while noise often appears as diffuse clouds or horizontal bands. Computer vision models are naturally optimized to detect these edges and textures.
  • Superiority of Masking: Unlike spectral subtraction, which can introduce "musical noise" artifacts, binary masking ensures that the signal within the identified regions remains untouched, leading to higher perceptual quality.
  • Cross-Domain Utility: The study demonstrates that "off-the-shelf" vision models like DeepLabV3 can be applied to audio tasks with minimal modification, provided the data is represented as a high-quality spectrogram.

Experimental Results

The researchers compared six segmentation models against three specialized audio denoising baselines: R-CED, Noise2Noise, and TS-U-Net.

The results were decisive. DeepLabV3 emerged as the top performer, achieving a Signal-to-Distortion Ratio (SDR) of 9.96. In contrast, the specialized audio-domain models (such as TS-U-Net) only reached an SDR of approximately 1.98. This represents a nearly five-fold improvement in denoising performance by adopting a visual approach.

Furthermore, the model demonstrated impressive generalization capabilities. Without retraining, the DVAD framework was successfully applied to human speech denoising and audio source separation (isolating individual birds in a multi-species recording).

Limitations and Future Directions

While highly effective, the DVAD approach faces challenges when noise sources perfectly overlap with the target signal in both time and frequency. In such cases, a binary mask may remove parts of the desired signal. Additionally, the STFT/ISTFT pipeline adds computational overhead that may impact real-time applications.

Future research will focus on developing audio-specific segmentation architectures designed to account for the unique physics of sound, such as harmonic structures and overtones, which are currently treated as independent visual features by standard vision models.

Summary updated

BibTeX
@inproceedings{zhang2023birdsoundsdenoising,
  author = {Youshan Zhang and Jialu Li},
  title = {{BirdSoundsDenoising: Deep Visual Audio Denoising for Bird Sounds}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00228},
}
Figure