A Hybrid Differential Evolution Approach to Designing Deep Convolutional Neural Networks for Image Classification

Bin Wang, Yanan Sun, Bing Xue, Mengjie ZhangAustralasian Conference on Artificial Intelligence 2018 · DOI · OpenReview

TL;DR

This paper introduces DECNN, a novel hybrid differential evolution algorithm designed to automatically discover optimal Convolutional Neural Network (CNN) architectures. By overcoming the limitations of fixed-length encoding and introducing new mutation and crossover operators, DECNN can evolve CNNs of variable depths and optimize their hyperparameters, achieving competitive results on various image classification benchmarks.

What Problem Does This Paper Address?

Designing effective Convolutional Neural Networks (CNNs) for image classification is a complex and time-consuming task. Manually crafting these architectures, including determining the number of layers, their types, and their specific parameters (like filter sizes and strides), requires significant expertise and extensive experimentation. While evolutionary computation (EC) methods have shown promise in automating this process, existing approaches often struggle with evolving variable-length architectures or optimizing hyperparameters efficiently. Specifically, some methods require a predefined maximum network depth, limiting their search space, while others fail to tune the crucial parameters within each layer. This paper aims to address these limitations by developing an EC algorithm that can freely evolve CNN architectures of any length and effectively optimize their internal configurations.

Methods

The core of the proposed method is a hybrid differential evolution (DE) algorithm named DECNN. The key innovations lie in three main areas:

  • Refined Encoding Scheme: The authors adapt an existing IP-Based Encoding Strategy (IPES) to handle variable-length CNN architectures. This refinement removes the constraint of a predefined maximum network depth, allowing the algorithm to explore a wider range of architectural possibilities. Each layer of the CNN is represented by an IP address-like structure, with attributes encoded into binary strings.
  • Variable-Length DE Operations: Standard DE algorithms are designed for fixed-length vectors. DECNN introduces new mutation and crossover operators specifically tailored to work with these variable-length encoded CNN architectures. This allows the algorithm to effectively manipulate and combine different architectural components.
  • Second Crossover Operator: A novel "second crossover" operator is introduced. This operator is designed to generate offspring with lengths that differ from their parents. By creating children that are either longer or shorter than their parents, this operator facilitates the exploration of optimal network depths, effectively searching for the best number of layers.

Datasets

The proposed DECNN algorithm was evaluated on six widely-used benchmark datasets:

  • MNIST Basic (MB): The standard MNIST dataset for handwritten digit recognition.
  • MNIST variants:
    • MNIST with a black and white image as the Background Image (MBI)
    • MNIST Digits Rotated with a black and white image as the Background Image (MDRBI)
    • MNIST with a Random Background (MRB)
    • MNIST with Rotated Digits (MRD)
  • CONVEX: A dataset consisting of convex and non-convex images.

These datasets were chosen to provide a comprehensive test of the algorithm's performance across different levels of complexity, including multi-class classification (MNIST variants) and binary classification (CONVEX).

Key Insights and Ideas

  • Variable-Length Architectures are Crucial: The paper strongly suggests that the ability to evolve CNNs of varying depths is essential for finding optimal architectures, as different tasks may benefit from shallower or deeper networks.
  • DE's Potential for CNN Architecture Search: While DE is known for global optimization, its application to evolving complex, variable-length structures like CNNs was underexplored. This work demonstrates its effectiveness in this domain.
  • The Power of a Targeted Crossover: The introduction of a second crossover operator specifically designed to alter offspring length proves to be a key mechanism for exploring architectural depth, a critical aspect of CNN design.
  • Encoding Matters: The refinement of the IP-Based Encoding Strategy is fundamental, enabling the DE operators to work with the variable-length nature of CNN architectures.

Key Results

The DECNN algorithm demonstrated competitive performance across the tested datasets:

  • State-of-the-Art Comparison: DECNN achieved strong results when compared to 12 state-of-the-art methods. It ranked first on the MBI and MRB datasets, second and third on MRD and MDRBI, respectively, and fifth on MB and CONVEX.
  • Superiority over IPPSO: In a direct comparison with IPPSO, a particle swarm optimization method using a similar encoding strategy, DECNN consistently outperformed IPPSO in terms of accuracy across all six datasets.
  • Best Results: DECNN achieved the best error rates on five out of the six datasets when compared to the 12 state-of-the-art methods, highlighting its effectiveness. For instance, it achieved error rates of 1.03% on MB, 5.67% on MBI, 32.85% on MDRBI, 3.46% on MRB, and 4.07% on MRD.

Limitations and Future Work

  • Computational Cost: While the paper mentions efforts to reduce computational cost (e.g., training on a partial dataset for a limited number of epochs), the fitness evaluation of CNN architectures remains inherently time-consuming.
  • Future Work: The authors suggest that DECNN's strong performance on complex datasets implies its potential for even larger and more intricate real-world applications. Further investigation on such datasets is recommended to assess its suitability for industrial use. Additionally, exploring further enhancements to the DE operations or encoding scheme could yield even better results.

Summary updated

Abstract

Convolutional Neural Networks (CNNs) have demonstrated their superiority in image classification, and evolutionary computation (EC) methods have recently been surging to automatically design the architectures of CNNs to save the tedious work of manually designing CNNs. In this paper, a new hybrid differential evolution (DE) algorithm with a newly added crossover operator is proposed to evolve the architectures of CNNs of any lengths, which is named DECNN. There are three new ideas in the proposed DECNN method. Firstly, an existing effective encoding scheme is refined to cater for variable-length CNN architectures; Secondly, the new mutation and crossover operators are developed for variable-length DE to optimise the hyperparameters of CNNs; Finally, the new second crossover is introduced to evolve the depth of the CNN architectures. The proposed algorithm is tested on six widely-used benchmark datasets and the results are compared to 12 state-of-the-art methods, which shows the proposed method is vigorously competitive to the state-of-the-art algorithms. Furthermore, the proposed method is also compared with a method using particle swarm optimisation with a similar encoding strategy named IPPSO, and the proposed DECNN outperforms IPPSO in terms of the accuracy.
BibTeX
@inproceedings{wang2018hybrid,
  author = {Bin Wang and Yanan Sun and Bing Xue and Mengjie Zhang},
  title = {{A Hybrid Differential Evolution Approach to Designing Deep Convolutional Neural Networks for Image Classification}},
  booktitle = {Australasian Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.26686/wgtn.13158293},
}
Figure