Research on improved convolutional wavelet neural network

Jingwei Liu, Peixuan Li, Xuehan Tang, Jiaxin Li, Jiaming ChenScientific 2021 · DOI

TL;DR

This paper introduces two novel neural network architectures, CWNN and WCNN, designed to overcome limitations in existing deep learning models like BPNN, RBFNN, and CNN. By integrating wavelet functions into convolutional neural networks, they achieve improved performance in terms of training speed, precision, and error reduction, particularly for image classification tasks.

What Problem Does This Paper Address?

Traditional artificial neural networks (ANNs) and deep learning models face several challenges. Backpropagation Neural Networks (BPNNs) are prone to getting stuck in local minima and have slow convergence. Radial Basis Function Neural Networks (RBFNNs) can be unstable and difficult to configure. Convolutional Neural Networks (CNNs), while powerful for feature extraction, can be computationally expensive for simpler tasks and may have limited maximum precision. This research aims to address these issues by proposing new network architectures that combine the strengths of wavelet transforms with convolutional and backpropagation principles to achieve better performance across various metrics.

Methods

The paper introduces several novel approaches:

  • Wavelet Neural Network (WNN): This network builds upon the structure of BPNN but uses a wavelet function as its activation function in the hidden layers. This is inspired by RBFNNs' use of radial basis functions and aims to solve the local minimum problem while maintaining faster training speeds than BPNN.
  • Convolutional Wavelet Neural Network (CWNN): This architecture modifies CNNs by replacing the fully connected layers (FCL) with a WNN. This aims to leverage the feature extraction capabilities of CNN's convolutional layers and the improved learning properties of WNN.
  • Wavelet Convolutional Neural Network (WCNN): This approach further enhances CNNs by replacing the activation function within the convolutional layers themselves with a wavelet transformation function, rather than the typical sigmoid function. This is applied within a Wavelet Convolutional Pooling Neural Network (WCPNN) component, which is then connected to a standard Fully Connected Neural Network (FCNN).

The authors also detail the gradient descent method used for training and the pseudocode for their WNN and CWNN algorithms.

Datasets

The research utilizes the following datasets for training and evaluation:

  • Custom Generated Dataset: A synthetic dataset with two features (x, y) and a label (z) generated using a sine and cosine function. This was used for initial "feasibility" and "hyperparameter optimization" experiments.
  • MNIST: A widely used dataset of handwritten digits, consisting of 60,000 images (50,000 for training, 10,000 for testing), with images resized to 28x28.
  • CIFAR-10: A dataset of 60,000 color images (32x32 pixels) in 10 classes (50,000 training, 10,000 testing), also resized to 28x28 for consistency.

Key Insights and Ideas

  • Wavelets as a Solution to Local Minima: The core idea is that wavelet functions, similar to radial basis functions, can help neural networks avoid getting stuck in local minima during training, a common issue with BPNNs.
  • Synergy of Convolution and Wavelets: By combining the feature extraction power of convolutional layers with the learning capabilities of wavelet-based networks, CWNN and WCNN aim to achieve superior performance.
  • Targeted Improvements: The paper systematically proposes improvements: WNN for general ANN issues, CWNN for CNN's fully connected layers, and WCNN for CNN's convolutional activation functions.
  • Empirical Validation: The authors conduct extensive simulations on standard datasets to rigorously compare their proposed models against existing ones.

Key Results

The paper presents several key findings from their experiments:

  • WNN Performance: WNN demonstrated better performance than BPNN and RBFNN, effectively solving the local minimum problem and showing improved training speed and precision. For instance, on a generated dataset, WNN's average training time was significantly lower (343 CTs) compared to BPNN (1305 CTs) and RBFNN (2630 CTs).
  • CWNN vs. CNN: CWNN consistently outperformed CNN on both MNIST and CIFAR-10 datasets. On MNIST, CWNN achieved a lower mean square error (0.14146 vs. 0.16431 for CNN) and a lower error rate (0.12915 vs. 0.16434 for CNN).
  • WCNN vs. CWNN: WCNN further improved upon CWNN. On MNIST, WCNN achieved a significantly lower mean square error (0.08850 vs. 0.14146 for CWNN) and a lower error rate (0.07723 vs. 0.18522 for CWNN).
  • Success Rates: In hyperparameter optimization experiments, WNN had the highest success rate (60%) for completing training within a set number of epochs, compared to RBFNN (40%) and BPNN (0%).

Limitations and Future Work

The authors acknowledge several limitations:

  • Convolutional Layer Limitations: Backpropagation algorithms, even within these new architectures, still rely on large datasets and can suffer from slow parameter adjustments in deep networks.
  • Pooling Layer Information Loss: The pooling layers in CNN-based architectures can discard valuable information.
  • Black Box Nature: Neural networks, including the proposed ones, remain largely "black boxes," making their internal workings difficult to interpret.

For future research, they suggest:

  • Improving Learning Speed and Ability: Exploring modifications to network structures to enhance learning efficiency.
  • Hierarchical Networks: Using WCNN and CWNN as building blocks for larger, more complex neural networks.
  • Further Validation: Conducting more experiments to confirm the feasibility and performance of their improved methods.

Summary updated

Abstract

Artificial neural networks (ANN) which include deep learning neural networks (DNN) have problems such as the local minimal problem of Back propagation neural network (BPNN), the unstable problem of Radial basis function neural network (RBFNN) and the limited maximum precision problem of Convolutional neural network (CNN). Performance (training speed, precision, etc.) of BPNN, RBFNN and CNN are expected to be improved. Main works are as follows: Firstly, based on existing BPNN and RBFNN, Wavelet neural network (WNN) is implemented in order to get better performance for further improving CNN. WNN adopts the network structure of BPNN in order to get faster training speed. WNN adopts the wavelet function as an activation function, whose form is similar to the radial basis function of RBFNN, in order to solve the local minimum problem. Secondly, WNN-based Convolutional wavelet neural network (CWNN) method is proposed, in which the fully connected layers (FCL) of CNN is replaced by WNN. Thirdly, comparative simulations based on MNIST and CIFAR-10 datasets among the discussed methods of BPNN, RBFNN, CNN and CWNN are implemented and analyzed. Fourthly, the wavelet-based Convolutional Neural Network (WCNN) is proposed, where the wavelet transformation is adopted as the activation function in Convolutional Pool Neural Network (CPNN) of CNN. Fifthly, simulations based on CWNN are implemented and analyzed on the MNIST dataset. Effects are as follows: Firstly, WNN can solve the problems of BPNN and RBFNN and have better performance. Secondly, the proposed CWNN can reduce the mean square error and the error rate of CNN, which means CWNN has better maximum precision than CNN. Thirdly, the proposed WCNN can reduce the mean square error and the error rate of CWNN, which means WCNN has better maximum precision than CWNN.
BibTeX
@inproceedings{liu2021research,
  author = {Jingwei Liu and Peixuan Li and Xuehan Tang and Jiaxin Li and Jiaming Chen},
  title = {{Research on improved convolutional wavelet neural network}},
  booktitle = {Scientific},
  year = {2021},
  doi = {10.1038/s41598-021-97195-6},
}
Figure