nnAudio: An on-the-Fly GPU Audio to Spectrogram Conversion Toolbox Using 1D Convolutional Neural Networks

Kin Wai Cheuk, Hans Anderson, Kat Agres, Dorien Herremans

TL;DR

This paper introduces nnAudio, a novel GPU-accelerated Python library for on-the-fly audio-to-spectrogram conversion using 1D convolutional neural networks. nnAudio significantly speeds up audio processing for deep learning tasks by enabling direct GPU computation and trainable spectrogram transformations, outperforming existing libraries in speed and flexibility.

What Problem Does This Paper Address?

Traditional audio processing for deep learning often involves a two-step pipeline: first, pre-computing and storing spectrograms from raw audio, and second, training a neural network on these pre-computed spectrograms. This approach faces several challenges:

  • Storage Demands: Spectrograms can consume vast amounts of disk space, especially when exploring various transformation parameters (e.g., different window sizes, frequency bins). For large datasets like MusicNet, this can lead to terabytes of storage requirements.
  • Inefficiency: Pre-computing spectrograms is time-consuming. If the optimal spectrogram representation isn't found, the entire pre-computation process needs to be repeated.
  • Lack of Trainability: Standard spectrogram transformations are fixed algorithms. This limits the ability to adapt the spectrogram generation process to the specific needs of a downstream neural network task.
  • CPU Bottlenecks: Many existing audio processing libraries rely on CPUs, which are significantly slower than GPUs for parallelizable tasks like spectrogram generation. Transferring data between CPU and GPU for model training adds further overhead.
  • Limited GPU Support: While some libraries offer GPU acceleration, they often lack support for crucial transformations like Constant-Q Transform (CQT) or don't integrate seamlessly with popular deep learning frameworks like PyTorch.

nnAudio aims to address these issues by providing a unified, GPU-accelerated, and trainable framework for audio-to-spectrogram conversion directly within the deep learning pipeline.

Methods

nnAudio leverages 1D convolutional neural networks (CNNs) implemented in PyTorch to perform various audio-to-spectrogram transformations. The core idea is to express these transformations as convolution operations, allowing them to be integrated as layers within a neural network.

  • 1D Convolution for DFT/STFT: The Discrete Fourier Transform (DFT) and Short-Time Fourier Transform (STFT) are implemented by using cosine and sine functions as filter kernels for 1D convolutions. This allows for efficient computation on GPUs.
  • Mel Spectrograms: Mel spectrograms are generated by applying a Mel filter bank kernel (initialized with weights from librosa) to the STFT output.
  • Constant-Q Transform (CQT): nnAudio implements multiple versions of CQT, including:
  • CQT1992: Based on Brown's 1992 algorithm, using frequency-domain CQT kernels.
  • CQT2010: Utilizes a recursive downsampling method for efficiency.
  • Improved Versions (CQT1992v2, CQT2010v2): These versions directly use time-domain CQT kernels, avoiding the computationally expensive conversion to frequency-domain kernels, leading to significant speedups.
  • Trainable Kernels: A key innovation is the ability to make the kernels (Fourier, Mel filter banks, CQT) trainable. This means these transformation parameters can be fine-tuned during the neural network training process, allowing for task-specific optimization of spectrogram generation.
  • On-the-Fly Processing: By integrating spectrogram generation as a neural network layer, nnAudio enables on-the-fly conversion of raw waveforms to spectrograms during model training, eliminating the need for pre-computation and storage.

Datasets

  • MAPS dataset: Used for benchmarking the speed of nnAudio against librosa. It consists of 1,770 .wav files.
  • MusicNet dataset: Used for evaluating the impact of different spectrogram representations on music transcription accuracy and for demonstrating the efficiency gains of nnAudio's on-the-fly processing.
  • Synthetically generated sine waves: Used in Section V-B to demonstrate the benefits of trainable kernels for a frequency prediction task.

Key Insights and Ideas

  • CNNs as Universal Spectrogram Generators: The paper demonstrates that 1D CNNs can effectively and efficiently implement a wide range of audio-to-spectrogram transformations, including STFT, Mel spectrograms, and various CQT variants.
  • Trainable Spectrograms for Task Adaptation: Making spectrogram generation layers trainable allows the model to learn optimal representations tailored to the specific downstream task, potentially leading to improved performance. This is a significant departure from fixed, traditional signal processing methods.
  • GPU Acceleration is Crucial for On-the-Fly Processing: The substantial speedups achieved by nnAudio are heavily reliant on GPU acceleration, making on-the-fly processing practical for large datasets and complex models.
  • Time-Domain CQT Kernels Offer Significant Speedup: The improved CQT implementations (CQT1992v2, CQT2010v2) that directly use time-domain kernels are notably faster than frequency-domain kernel approaches, especially for longer audio segments.
  • On-the-Fly Processing Outperforms Pre-processing for Exploration: While pre-processed spectrograms can be faster for inference once generated, nnAudio's on-the-fly approach drastically reduces the time and effort required for exploring different spectrogram parameters during model development.

Key Results

  • Speed Improvements:
    • nnAudio significantly reduces spectrogram extraction time compared to librosa. For example, converting 1,770 waveforms to spectrograms took 10.6 seconds with librosa but only 0.001 seconds with nnAudio for STFT on a Tesla v100 GPU.
  • Similar speedups were observed for Mel spectrograms (18.3s vs. 0.015s) and CQT (103.4s vs. 0.258s).
  • For the MusicNet dataset, nnAudio reduced the time to generate 34 types of spectrograms from 983 minutes to 99 minutes compared to the traditional approach.
  • Performance Comparison: nnAudio outperforms other GPU libraries like Kapre and torchaudio in terms of processing speed for STFT and MelSpec.
  • Accuracy: The paper shows that trainable kernels can lead to improved accuracy in tasks like frequency prediction, with models using trainable kernels consistently outperforming those with fixed kernels.
  • Output Quality: Qualitative comparisons (Figures 12-14) show that nnAudio's outputs are very similar to librosa's, with CQT1992v2 yielding the smoothest and best results among the CQT implementations.

Limitations and Future Work

  • Initialization Time: While processing is fast, there's an initial overhead for transferring kernels from RAM to GPU memory, which can take a few seconds. This is considered part of model initialization.
  • CQT Kernel Size: For very low frequencies and long audio segments, CQT kernels can become extremely long, potentially impacting performance. The paper mentions that using FFT instead of DFT for the initial stage of CQT might be explored in future work.
  • Aliasing in CQT: The downsampling method used in some CQT implementations can introduce aliasing, which is observable in logarithmic magnitude plots. Further study is needed to determine the effects of this aliasing on neural network models.
  • Future Work: The authors suggest exploring FFT-based CQT implementations and further investigating the impact of trainable kernels on various audio tasks. They also mention that their proposed framework is the first neural network-based toolbox to support CQT.

Summary updated

Abstract

In this paper, we present nnAudio, a new neural network-based audio processing framework with graphics processing unit (GPU) support that leverages 1D convolutional neural networks to perform time domain to frequency domain conversion. It allows on-the-fly spectrogram extraction due to its fast speed, without the need to store any spectrograms on the disk. Moreover, this approach also allows back-propagation on the waveforms-to-spectrograms transformation layer, and hence, the transformation process can be made trainable, further optimizing the waveform-to-spectrogram transformation for the specific task that the neural network is trained on. All spectrogram implementations scale as Big-O of linear time with respect to the input length. nnAudio, however, leverages the compute unified device architecture (CUDA) of 1D convolutional neural network from PyTorch, its short-time Fourier transform (STFT), Mel spectrogram, and constant-Q transform (CQT) implementations are an order of magnitude faster than other implementations using only the central processing unit (CPU). We tested our framework on three different machines with NVIDIA GPUs, and our framework significantly reduces the spectrogram extraction time from the order of seconds (using a popular python library librosa) to the order of milliseconds, given that the audio recordings are of the same length. When applying nnAudio to variable input audio lengths, an average of 11.5 hours are required to extract 34 spectrogram types with different parameters from the MusicNet dataset using librosa. An average of 2.8 hours is required for nnAudio, which is still four times faster than librosa. Our proposed framework also outperforms existing GPU processing libraries such as Kapre and torchaudio in terms of processing speed.
BibTeX
@article{cheuk2020nnaudio,
  author = {Kin Wai Cheuk and Hans Anderson and Kat Agres and Dorien Herremans},
  title = {{nnAudio: An on-the-Fly GPU Audio to Spectrogram Conversion Toolbox Using 1D Convolutional Neural Networks}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3019084},
}

Loading related content...

Figure