Fractional-Order Deep Backpropagation Neural Network

Yi Zhang, Chunhui Bao, Yifei Pu, Yi Zhang 0018

TL;DR

This paper introduces a novel "fractional-order deep backpropagation neural network" that leverages fractional calculus to potentially enhance learning capabilities and combat overfitting. By extending the order of derivatives in the backpropagation algorithm beyond integers and incorporating L₂ regularization, the authors demonstrate improved performance and deterministic convergence on the MNIST dataset.

What Problem Does This Paper Address?

Traditional artificial neural networks (ANNs), particularly deep learning models, have achieved remarkable success in various fields. However, they often suffer from overfitting, especially when dealing with limited training data. This phenomenon leads to high accuracy on the training set but poor generalization to unseen data. While backpropagation (BP) is a standard training algorithm, existing fractional-order BP networks have limitations, such as restricting the fractional order to a narrow range (0, 1] and not effectively addressing overfitting. This paper aims to overcome these limitations by proposing a more flexible fractional-order deep BP network with L₂ regularization.

Methods

The core innovation lies in extending the backpropagation algorithm to use fractional-order derivatives. Specifically, the authors employ the Caputo definition of fractional derivatives, which aligns with integer-order calculus for constant functions. The network architecture is a deep, multi-layer feedforward network.

Key methodological aspects include:

  • Fractional-Order Backpropagation: Instead of integer-order derivatives, the gradient descent update rules for weights are modified to use fractional-order derivatives. This is achieved by applying the fractional gradient descent method with the Caputo derivative.
  • L₂ Regularization: To combat overfitting, an L₂ regularization term is added to the total error function. This penalizes large weights, encouraging simpler models.
  • Fractional-Order Variational Method: This technique is used to analyze the influence of L₂ regularization on the convergence of the proposed network.
  • Convergence Analysis: The paper outlines four necessary conditions for the convergence of BP neural networks and then analyzes how the fractional-order approach and L₂ regularization affect these conditions.

Datasets

  • MNIST Handwritten Digit Dataset: This is the primary dataset used for training and evaluating the proposed fractional-order deep BP neural network. The dataset consists of 28x28 pixel grayscale images of handwritten digits. For experimental purposes, each image was divided into four 14x14 parts, vectorized, and the labels were one-hot encoded.

Key Insights and Ideas

  • Fractional Calculus for Enhanced Learning: The paper suggests that fractional calculus, with its inherent "long-term memory" and "nonlocality" properties, can offer advantages in neural network learning beyond traditional integer-order calculus.
  • Flexible Fractional Order: Unlike previous work, this model allows the fractional order (v) to be any positive real number, offering greater flexibility and potential for optimization.
  • L₂ Regularization's Crucial Role: The inclusion of L₂ regularization is shown to be vital for improving generalization and preventing overfitting, especially when training data is scarce.
  • Deterministic Convergence: The authors provide evidence and analysis suggesting that their proposed fractional-order BP network with L₂ regularization exhibits deterministic convergence.
  • Optimal Fractional Orders: Experiments reveal that there are specific fractional orders (around 10/9 and 11/9) that yield optimal performance for the MNIST dataset, indicating that the choice of fractional order is not arbitrary but can be tuned.

Key Results

  • Improved Accuracy and Reduced Overfitting: The proposed fractional-order deep BP neural network with L₂ regularization consistently achieved higher testing accuracies compared to traditional BP networks and fractional-order BP networks without regularization. For instance, with 60,000 training samples, the fractional-order BP with L₂ regularization achieved a testing accuracy of 96.70%, a significant improvement over the integer-order BP (95.00%) and fractional-order BP without regularization (95.00%).
  • Effectiveness with Small Datasets: The benefits of L₂ regularization were particularly pronounced when the training set size was small, demonstrating its effectiveness in mitigating overfitting in such scenarios.
  • Deterministic Convergence Verified: Figures 4 and 5 show that the total error (EL2) and its fractional-order derivative converged quickly and stably to near zero, supporting the claim of deterministic convergence.
  • Optimal Order Range: The study found that fractional orders greater than 1.6 led to a rapid decline in performance, and orders greater than 2 were significantly poorer than those in the 0 < v < 2 range. Optimal performance was observed around v = 10/9 and 11/9.

Limitations and Future Work

  • Limitations:
    • The paper notes that when the fractional order v equals 2, the change in weights becomes zero, preventing network updates. This implies a constraint on the fractional order, although the paper explores orders beyond 1.
  • While the paper explores a range of fractional orders, the optimal orders found were specific to the MNIST dataset. Generalizing these optimal orders to other datasets might require further investigation.
  • Future Work:
    • The authors suggest that a promising direction for future research is to explore the application of fractional calculus to other, more complex artificial neural network architectures.
  • Investigating the theoretical underpinnings and practical implications of fractional calculus in other machine learning domains could also be a valuable avenue.

Summary updated

Abstract

In recent years, the research of artificial neural networks based on fractional calculus has attracted much attention. In this paper, we proposed a fractional-order deep backpropagation (BP) neural network model with <svg xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" style="vertical-align:-3.1815pt" id="M1" height="11.8174pt" version="1.1" viewBox="-0.0498162 -8.6359 13.3712 11.8174" width="13.3712pt"><g transform="matrix(.013,0,0,-0.013,0,0)"><path id="g113-77" d="M541 160L512 170C485 116 462 88 439 67C411 41 376 35 318 35C272 35 238 37 224 48C207 61 205 85 212 121L290 533C305 611 308 615 391 622L397 650H139L133 622C217 615 221 612 206 533L126 118C111 41 103 34 23 26L17 0H474C489 31 528 124 541 160Z"/></g><g transform="matrix(.0091,0,0,-0.0091,8.294,3.132)"><path id="g50-51" d="M414 144C384 79 371 75 317 75H135L276 221C367 316 408 376 408 465C408 570 327 635 237 635C179 635 131 609 100 575L42 494L67 471C94 510 138 565 205 565C277 565 321 517 321 435C321 348 258 270 195 195C146 137 88 81 33 26V0H411C423 44 433 88 446 135L414 144Z"/></g></svg> regularization. The proposed network was optimized by the fractional gradient descent method with Caputo derivative. We also illustrated the necessary conditions for the convergence of the proposed network. The influence of <svg xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" style="vertical-align:-3.1815pt" id="M2" height="11.8174pt" version="1.1" viewBox="-0.0498162 -8.6359 13.3712 11.8174" width="13.3712pt"><g transform="matrix(.013,0,0,-0.013,0,0)"><use xlink:href="#g113-77"/></g><g transform="matrix(.0091,0,0,-0.0091,8.294,3.132)"><use xlink:href="#g50-51"/></g></svg> regularization on the convergence was analyzed with the fractional-order variational method. The experiments have been performed on the MNIST dataset to demonstrate that the proposed network was deterministically convergent and can effectively avoid overfitting.
BibTeX
@article{zhang2018fractional,
  author = {Yi Zhang and Chunhui Bao and Yifei Pu and Yi Zhang},
  title = {{Fractional-Order Deep Backpropagation Neural Network}},
  journal = {Comput. Intell. Neurosci},
  year = {2018},
  doi = {10.1155/2018/7361628},
}
Figure