TL;DR
This paper surveys eight recent works that successfully fuse Evolutionary Computation (EC) with Machine Learning (ML) and Deep Learning (DL). The core contribution is demonstrating how these fields, when combined judiciously, create a powerful alliance that pushes the boundaries of AI, leading to novel solutions for classification, neuroevolution, and adversarial attacks.
What Problem Does This Paper Address?
The paper addresses the growing need to combine the strengths of Evolutionary Computation (EC) and Machine Learning (ML), particularly Deep Learning (DL). While ML excels at pattern recognition and prediction, it often relies on gradients and can struggle with design problems or situations with limited prior knowledge. EC, on the other hand, offers robust optimization, the ability to design novel solutions from scratch, and flexibility in handling complex objectives. The challenge is to effectively bridge these two paradigms to leverage their complementary advantages and tackle more complex AI problems.
Methods
The paper surveys eight distinct projects, each showcasing a unique blend of EC and ML/DL. Key methods include:
- Evolutionary Symbolic Regression: Using EC to evolve symbolic expressions for classification tasks.
- Ensemble Learning: Developing novel ensemble algorithms (Classy Ensemble) that combine multiple models through weighted aggregation, often using evolutionary techniques for model selection.
- EC Toolkits: Creating comprehensive, Python-based toolkits (EC-KitY) that facilitate the integration of EC with ML libraries like scikit-learn.
- Neuroevolution: Evolving components of neural networks, such as activation functions, or optimizing entire network architectures using EC.
- Adversarial ML: Employing EC for gradient-free, query-efficient, black-box attacks on deep networks, generating adversarial examples and patches to fool object detectors. This also includes using EC to manipulate model explanations.
Datasets
The paper mentions the use of various datasets across its surveyed works:
- Classification: "over 162 datasets" and "over 153 machine learning datasets" are mentioned for classification tasks.
- Image Classification: MNIST, FashionMNIST, KMNIST, and USPS are explicitly listed for evolving activation functions.
- Object Detection: ImageNet and CIFAR10 are mentioned in the context of adversarial attacks.
No new benchmarks are introduced; rather, existing, well-known datasets are used to evaluate the proposed methods.
Key Insights and Ideas
- Synergistic Power: The overarching insight is that EC and ML/DL are not just compatible but mutually reinforcing. EC can optimize complex search spaces and design novel structures, while ML/DL provides powerful predictive models and feature extraction capabilities.
- Gradient-Free Optimization: EC's ability to perform optimization without requiring gradients is crucial for black-box adversarial attacks, where model internals are inaccessible.
- Evolving Network Components: Evolving activation functions or entire network architectures (neuroevolution) offers a more flexible and potentially more performant alternative to manually designed deep learning models.
- Robustness and Security: The research highlights how EC can be used to both attack and potentially defend against adversarial examples, a critical area for the security of AI systems.
- Interpretability Manipulation: EC can be used to manipulate the explanations generated by AI systems, raising important questions about trust and transparency in AI.
Key Results
- Competitive Performance: The evolutionary symbolic regression classifiers were found to be competitive with state-of-the-art ML algorithms like XGBoost and LightGBM.
- Improved Adversarial Attacks: The proposed gradient-free, query-efficient evolutionary attack methods demonstrated effectiveness against various deep learning models and datasets, including ImageNet, CIFAR10, and MNIST.
- Realistic Adversarial Patches: The "Patch of Invisibility" work showed that physically plausible adversarial patches can be generated using EC, successfully fooling object detectors in real-world scenarios.
- Effective Neuroevolution: The adaptive combination of GA and Novelty Search for deep neuroevolution outperformed novelty search alone and was on par with or better than standard GAs in certain scenarios.
- Usability of EC-KitY: The EC-KitY toolkit proved effective in a university course, enabling students to tackle diverse complex problems.
Limitations and Future Work
- Computational Cost: While EC can reduce training time in some cases, evolutionary algorithms can still be computationally intensive, especially for large search spaces.
- Hyperparameter Tuning: Many EC methods, like other ML techniques, require careful tuning of hyperparameters (e.g., population size, mutation rates).
- Adaptability of Adversarial Methods: While the adversarial attack methods are shown to be effective, their adaptation to targeted attacks or different network architectures is noted as a potential area for future work.
- Expanding the Alliance: The authors express a strong desire to continue expanding the successful joint ventures between EC and ML/DL, suggesting ongoing research and development in this area.
Summary updated