Spectral Domain Convolutional Neural Network

Fang Liu, Bochen Guan, Jinnian Zhang, William A. Sethares, Richard Kijowski, Fang Liu 0005ICASSP 2021 · DOI

TL;DR

This paper introduces SpecNet, a novel Convolutional Neural Network (CNN) architecture that performs convolutions and activations in the spectral domain. By exploiting the spectral concentration of feature map energy and applying a thresholding technique, SpecNet significantly reduces memory consumption (up to 60%) with minimal impact on accuracy, making deep learning more accessible on resource-constrained devices.

What Problem Does This Paper Address?

Training deep Convolutional Neural Networks (CNNs) is notoriously memory-intensive, posing a significant challenge for modern GPUs with limited memory, embedded systems, and mobile devices. While previous research has focused on accelerating training or compressing model weights, this paper identifies the intermediate layer outputs, known as feature maps, as the primary bottleneck for memory consumption. The authors observe that, similar to natural images, these feature maps tend to have their energy concentrated in the spectral domain. This observation forms the basis for their work, aiming to directly address the memory footprint of feature maps.

Methods

The core innovation of SpecNet lies in its spectral domain processing. Instead of performing convolutions in the spatial domain, SpecNet transforms both the input feature maps and the kernels into the spectral domain using the Fast Fourier Transform (FFT). The convolution operation then becomes a simple element-wise multiplication in the spectral domain.

A key technique employed is spectral domain compression. After the spectral convolution, a configurable threshold ($\beta$) is applied to the resulting feature maps. Values below this threshold are set to zero, creating sparse feature maps. This sparsity allows for efficient storage by only retaining non-zero entries, thus reducing memory usage.

SpecNet also introduces a novel activation function designed for spectral domain operations. This function is crafted to preserve the symmetry properties of spectral feature maps, ensuring that when transformed back to the spatial domain, they remain real-valued without generating spurious phases. The activation function is designed to be computationally inexpensive and allow for efficient gradient calculation during backpropagation.

Datasets

The paper evaluates SpecNet on the following datasets:

  • CIFAR-10
  • SVHN
  • ImageNet

Key Insights and Ideas

  • Feature maps are spectrally concentrated: The paper's central insight is that feature maps, like natural images, exhibit concentrated energy in the spectral domain. This property is the foundation for spectral domain compression.
  • Spectral domain offers memory reduction: By performing convolutions and activations in the spectral domain and applying thresholding, SpecNet can achieve significant memory savings by storing sparse feature maps.
  • Sparsity for memory efficiency: The thresholding mechanism directly targets memory reduction by zeroing out low-energy spectral components, enabling sparse storage.
  • Symmetry-preserving activation: A custom activation function is crucial for maintaining the real-valued nature of spatial feature maps after spectral transformations, preventing artifacts.
  • Trade-off between memory and accuracy: Increasing the spectral compression threshold ($\beta$) leads to greater memory savings but can introduce approximation errors, potentially impacting accuracy. This highlights a tunable trade-off.

Key Results

SpecNet demonstrates significant memory reduction across various CNN architectures and datasets:

  • Memory Reduction: SpecNet can reduce memory consumption by approximately 60% without a significant loss of performance. For instance, on ImageNet, SpecNet implementations of AlexNet, VGG, and DenseNet consumed less than half the memory of their original counterparts.
  • Accuracy: While there is a slight decrease in top-1 accuracy compared to original models, the top-5 accuracy remains largely unaffected (often above 95% for $\beta < 1.0$). This indicates a favorable trade-off between memory efficiency and performance.
  • Training Convergence: SpecNet implementations show comparable training convergence rates to their spatial domain counterparts, suggesting that the spectral domain processing does not hinder the learning process.
  • Outperforming other methods: On CIFAR-10, SpecNet achieved the lowest memory usage among several compared memory-efficient algorithms while maintaining high testing accuracy.

Limitations and Future Work

  • Approximation Error: The paper acknowledges that storing feature maps sparsely in the spectral domain introduces approximation errors during backward propagation, which can affect convergence rate or accuracy.
  • Hyper-parameter Tuning: The performance of SpecNet is sensitive to the choice of the spectral compression threshold ($\beta$). Further exploration of hyper-parameter settings could potentially improve performance.
  • Future Work: The authors suggest merging SpecNet with other memory-saving techniques, such as model compression (quantization, pruning) and scheduling, to achieve even greater memory efficiency. They also note that the spectral domain processing might incur additional memory costs due to CUDA implementation details.

Summary updated

BibTeX
@inproceedings{liu2021spectral,
  author = {Fang Liu and Bochen Guan and Jinnian Zhang and William A. Sethares and Richard Kijowski and Fang Liu},
  title = {{Spectral Domain Convolutional Neural Network}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9413409},
}
Figure