Abstract
Neural Architecture Search (NAS) automates neural network design, reducing dependence on human expertise. While NAS methods are computationally intensive and dataset-specific, auxiliary predictors reduce the models needing training, decreasing search time. This strategy is used to generate architectures satisfying multiple computational constraints. Recently, Transferable NAS has emerged, generalizing the search process from dataset-dependent to task-dependent. In this field, DiffusionNAG is a state-of-the-art method. This diffusion-based approach streamlines computation, generating architectures optimized for accuracy on unseen datasets without further adaptation. However, by focusing solely on accuracy, DiffusionNAG overlooks other crucial objectives like model complexity, computational efficiency, and inference latency -- factors essential for deploying models in resource-constrained environments. This paper introduces the Pareto-Optimal Many-Objective Neural Architecture Generator (POMONAG), extending DiffusionNAG via a many-objective diffusion process. POMONAG simultaneously considers accuracy, number of parameters, multiply-accumulate operations (MACs), and inference latency. It integrates Performance Predictor models to estimate these metrics and guide diffusion gradients. POMONAG's optimization is enhanced by expanding its training Meta-Dataset, applying Pareto Front Filtering, and refining embeddings for conditional generation. These enhancements enable POMONAG to generate Pareto-optimal architectures that outperform the previous state-of-the-art in performance and efficiency. Results were validated on two search spaces -- NASBench201 and MobileNetV3 -- and evaluated across 15 image classification datasets.
Introduction
Designing optimal neural network architectures is traditionally a manual, trial-and-error process. While Neural Architecture Search (NAS) was developed to automate this, early methods were often computationally prohibitive and limited to specific datasets. The emergence of "Transferable NAS," such as the diffusion-based DiffusionNAG, addressed generalization and speed by generating architectures for unseen tasks. However, these methods primarily optimized for accuracy, neglecting critical real-world constraints like model size, computational cost (MACs), and inference latency.
POMONAG addresses this gap by enabling a diffusion-based generator to navigate the complex trade-offs between performance and efficiency, producing architectures suitable for deployment in resource-constrained environments.
Methodology
POMONAG extends the diffusion-based generative framework by introducing several key innovations to handle multiple objectives simultaneously.
Many-Objective Reverse Diffusion Guidance
The core of POMONAG is a denoising process guided by four specialized Performance Predictors ($f_\phi, f_\pi, f_\mu, f_\lambda$). These predictors estimate accuracy, parameter count, MACs, and latency, respectively. Their gradients steer the architecture generation toward the "Optimal Region" of the search space, balancing multiple competing objectives.
Pareto Front Filtering and Stretching
To ensure a diverse set of high-quality architectures, POMONAG employs Pareto Front Filtering to discard sub-optimal designs. A "stretching" technique is applied to the generated batch to cover a broader range of the efficiency-performance trade-off spectrum.
Enhanced Meta-Datasets
The training pipeline was improved by expanding the meta-dataset to 20,000 architecture-task pairs and upgrading the dataset encoder to a Vision Transformer (ViT-B-16). This allows the model to better capture task-specific nuances during conditional generation.
Experimental Results
POMONAG was evaluated across 15 image classification datasets using the NASBench201 and MobileNetV3 search spaces. The framework offers three variants: Accurate (maximizing accuracy), Efficient (minimizing size/latency), and Balanced (optimal trade-off).
Performance and Efficiency
On the MobileNetV3 search space, the POMONAG_Acc variant consistently outperformed competitors like DiffusionNAG and TNAS. For instance, it achieved 87.62% accuracy on the Aircraft dataset, a significant improvement over the previous state-of-the-art of 82.31%.
In the NASBench201 space, the POMONAG_Eff variant demonstrated massive efficiency gains, reducing parameters by approximately 90%, MACs by 93%, and latency by 65% compared to DiffusionNAG, while maintaining competitive accuracy levels.
Search Efficiency
A key advantage of POMONAG is its "one-shot" transfer capability. Unlike many NAS methods that require training dozens or hundreds of candidates, POMONAG typically identifies an optimal model by training only a single architecture per dataset. This efficiency is driven by the high quality of the underlying performance predictors, which significantly outperform previous baselines.
Conclusion
POMONAG demonstrates that diffusion models, when guided by multi-objective predictors, are highly effective at navigating complex architectural search spaces. By treating NAS as a generative task, POMONAG produces Pareto-optimal architectures that significantly outperform previous methods in both performance and efficiency. Future work will focus on extending this framework to other computer vision tasks such as object detection and segmentation.
Summary updated