Abstract
LEAF (Learning Evolutionary AI Framework) is an AutoML system that uses evolutionary algorithms to simultaneously optimize neural network architectures, hyperparameters, and model size. By co-evolving "blueprints" and "modules," LEAF can discover state-of-the-art deep learning models that are both more accurate and more computationally efficient than those designed by human experts or existing commercial AutoML tools.
Introduction
The design of deep neural networks (DNNs) has traditionally been a labor-intensive process of trial and error, requiring significant domain expertise. While automated machine learning (AutoML) tools have emerged to alleviate this burden, most existing solutions focus narrowly on hyperparameter optimization—tuning parameters like learning rate while keeping the network architecture static. This approach fails to address the "architecture bottleneck," where the arrangement of layers often has a more profound impact on performance than parameter tuning. Furthermore, the increasing demand for high-performance models on resource-constrained edge devices necessitates a focus on computational efficiency.
The Learning Evolutionary AI Framework (LEAF) addresses these challenges by employing evolutionary algorithms to simultaneously optimize neural network architectures, hyperparameters, and model size.
Methodology: CoDeepNEAT and Hierarchical Evolution
At the core of LEAF is CoDeepNEAT, an extension of the NeuroEvolution of Augmenting Topologies (NEAT) algorithm designed for the scale of deep learning. CoDeepNEAT introduces a hierarchical co-evolutionary approach that mimics the modular design patterns used by human researchers (e.g., the blocks in ResNet or Inception).
The system maintains two distinct populations:
- Modules: Small sub-networks consisting of specific sequences of layers (e.g., convolutional or LSTM layers).
- Blueprints: Graphs that define how these modules are interconnected to form a complete network.
This separation allows the framework to discover complex structural patterns through the reuse of optimized modules. To manage the computational intensity of training numerous candidate networks, LEAF utilizes a distributed system layer (StudioML) to parallelize training across cloud GPU clusters.
Experimental Evaluation
LEAF was evaluated on two challenging benchmarks:
- Wikidetox: A natural language processing task involving the binary classification of 160,000 Wikipedia comments for toxicity.
- Chest X-ray8: A multi-task image classification benchmark with over 112,000 high-resolution images labeled with 14 lung diseases.
Multiobjective Optimization
Beyond accuracy, LEAF employs Pareto-based multiobjective optimization. This allows the system to find a "frontier" of models that balance performance against secondary objectives, such as minimizing the total number of parameters.
Key Results and Discussion
Experimental results demonstrate that LEAF consistently outperforms both human-designed models and leading commercial AutoML platforms.
Performance Benchmarks
On the Wikidetox dataset, LEAF discovered an architecture that surpassed the Kaggle competition winner’s hand-designed model. As shown in Figure 4, LEAF achieved a lower toxicity error than Google AutoML, Microsoft TLC, and Yelp’s MOE/SigOpt, eventually reaching a performance level comparable to the best known results.
On the Chest X-ray8 task, LEAF reached an AUROC of 84.3%, nearly matching the state-of-the-art hand-designed CheXNet (84.4%) and significantly exceeding Google AutoML’s 79.7%.
Structural Regularization and Efficiency
A key insight from the study is that optimizing for smaller network sizes often improves generalization. This suggests that the pressure to reuse modules acts as a form of structural regularization. For instance, LEAF identified a model with only 56,000 parameters that performed nearly as well as one with 125,000 parameters, demonstrating significant potential for model compression during the architecture search phase.
Conclusion
LEAF represents a significant step toward fully automated, high-performance AI development. By moving beyond simple hyperparameter tuning to the co-evolution of complex architectures, it can discover design patterns that rival or exceed human expertise. While the computational cost remains high—requiring thousands of CPU hours for complex tasks—the ability to automate the discovery of efficient, state-of-the-art models provides a compelling alternative to manual design. Future work will focus on reducing these costs and expanding LEAF's application to domains such as robotics and reinforcement learning.
Summary updated