TL;DR
This paper introduces two novel neural network architectures, CWNN and WCNN, designed to overcome limitations in existing deep learning models like BPNN, RBFNN, and CNN. By integrating wavelet functions into convolutional neural networks, they achieve improved performance in terms of training speed, precision, and error reduction, particularly for image classification tasks.
What Problem Does This Paper Address?
Traditional artificial neural networks (ANNs) and deep learning models face several challenges. Backpropagation Neural Networks (BPNNs) are prone to getting stuck in local minima and have slow convergence. Radial Basis Function Neural Networks (RBFNNs) can be unstable and difficult to configure. Convolutional Neural Networks (CNNs), while powerful for feature extraction, can be computationally expensive for simpler tasks and may have limited maximum precision. This research aims to address these issues by proposing new network architectures that combine the strengths of wavelet transforms with convolutional and backpropagation principles to achieve better performance across various metrics.
Methods
The paper introduces several novel approaches:
- Wavelet Neural Network (WNN): This network builds upon the structure of BPNN but uses a wavelet function as its activation function in the hidden layers. This is inspired by RBFNNs' use of radial basis functions and aims to solve the local minimum problem while maintaining faster training speeds than BPNN.
- Convolutional Wavelet Neural Network (CWNN): This architecture modifies CNNs by replacing the fully connected layers (FCL) with a WNN. This aims to leverage the feature extraction capabilities of CNN's convolutional layers and the improved learning properties of WNN.
- Wavelet Convolutional Neural Network (WCNN): This approach further enhances CNNs by replacing the activation function within the convolutional layers themselves with a wavelet transformation function, rather than the typical sigmoid function. This is applied within a Wavelet Convolutional Pooling Neural Network (WCPNN) component, which is then connected to a standard Fully Connected Neural Network (FCNN).
The authors also detail the gradient descent method used for training and the pseudocode for their WNN and CWNN algorithms.
Datasets
The research utilizes the following datasets for training and evaluation:
- Custom Generated Dataset: A synthetic dataset with two features (x, y) and a label (z) generated using a sine and cosine function. This was used for initial "feasibility" and "hyperparameter optimization" experiments.
- MNIST: A widely used dataset of handwritten digits, consisting of 60,000 images (50,000 for training, 10,000 for testing), with images resized to 28x28.
- CIFAR-10: A dataset of 60,000 color images (32x32 pixels) in 10 classes (50,000 training, 10,000 testing), also resized to 28x28 for consistency.
Key Insights and Ideas
- Wavelets as a Solution to Local Minima: The core idea is that wavelet functions, similar to radial basis functions, can help neural networks avoid getting stuck in local minima during training, a common issue with BPNNs.
- Synergy of Convolution and Wavelets: By combining the feature extraction power of convolutional layers with the learning capabilities of wavelet-based networks, CWNN and WCNN aim to achieve superior performance.
- Targeted Improvements: The paper systematically proposes improvements: WNN for general ANN issues, CWNN for CNN's fully connected layers, and WCNN for CNN's convolutional activation functions.
- Empirical Validation: The authors conduct extensive simulations on standard datasets to rigorously compare their proposed models against existing ones.
Key Results
The paper presents several key findings from their experiments:
- WNN Performance: WNN demonstrated better performance than BPNN and RBFNN, effectively solving the local minimum problem and showing improved training speed and precision. For instance, on a generated dataset, WNN's average training time was significantly lower (343 CTs) compared to BPNN (1305 CTs) and RBFNN (2630 CTs).
- CWNN vs. CNN: CWNN consistently outperformed CNN on both MNIST and CIFAR-10 datasets. On MNIST, CWNN achieved a lower mean square error (0.14146 vs. 0.16431 for CNN) and a lower error rate (0.12915 vs. 0.16434 for CNN).
- WCNN vs. CWNN: WCNN further improved upon CWNN. On MNIST, WCNN achieved a significantly lower mean square error (0.08850 vs. 0.14146 for CWNN) and a lower error rate (0.07723 vs. 0.18522 for CWNN).
- Success Rates: In hyperparameter optimization experiments, WNN had the highest success rate (60%) for completing training within a set number of epochs, compared to RBFNN (40%) and BPNN (0%).
Limitations and Future Work
The authors acknowledge several limitations:
- Convolutional Layer Limitations: Backpropagation algorithms, even within these new architectures, still rely on large datasets and can suffer from slow parameter adjustments in deep networks.
- Pooling Layer Information Loss: The pooling layers in CNN-based architectures can discard valuable information.
- Black Box Nature: Neural networks, including the proposed ones, remain largely "black boxes," making their internal workings difficult to interpret.
For future research, they suggest:
- Improving Learning Speed and Ability: Exploring modifications to network structures to enhance learning efficiency.
- Hierarchical Networks: Using WCNN and CWNN as building blocks for larger, more complex neural networks.
- Further Validation: Conducting more experiments to confirm the feasibility and performance of their improved methods.
Summary updated