TL;DR
POMONAG extends diffusion-based transferable Neural Architecture Search (NAS) into a true many-objective generator. By guiding reverse diffusion with multiple specialized predictors, it simultaneously optimizes models for accuracy, parameter count, MACs, and inference latency on unseen datasets—yielding Pareto-optimal architectures while requiring only a single network to be trained.
What Problem Does This Paper Address?
Traditional Neural Architecture Search (NAS) can discover high-performing deep networks, but typically demands colossal compute budgets and operates on a single target dataset from scratch. Transferable NAS methods, like DiffusionNAG, improved on this by using graph diffusion models to generate dataset-conditioned architectures for unseen tasks without retraining the generator.
However, DiffusionNAG focuses solely on accuracy. In practical deployments—such as edge computing and mobile devices—accuracy cannot come at the expense of memory footprint, compute operations, or latency. Existing multi-objective NAS approaches either scale poorly, require training dozens to hundreds of candidates, or struggle to navigate complex trade-off spaces across unseen tasks.
Methods
POMONAG builds upon the graph-diffusion framework of DiffusionNAG and introduces several key innovations:
- Many-Objective Reverse Diffusion Guidance: Instead of guiding the diffusion trajectory with only an accuracy predictor, POMONAG incorporates four simultaneous auxiliary predictors (Accuracy $f_\phi$, Parameters $f_\pi$, MACs $f_\mu$, and Latency $f_\lambda$). Their gradients collectively steer the reverse diffusion process toward high-density regions of Pareto-optimal designs.
- Pareto Front Stretching: Rather than generating architectures under a static compromise, POMONAG generates candidates in two targeted batches of 128: one biased toward strict efficiency and another toward maximal accuracy. The weighting factors for each predictor are dynamically optimized per search space using Optuna.
- Pareto Front Filtering: After generation and structure validation, a 2D Pareto front is constructed for each secondary metric. From these, users can directly select three designated profiles without training the pool:
- $\text{POMONAG}_{\text{Acc}}$: Highest predicted accuracy.
- $\text{POMONAG}_{\text{Bal}}$: Best ratio of accuracy to the efficiency metric.
- $\text{POMONAG}_{\text{Eff}}$: Minimum resource footprint / maximum efficiency.
- Upgraded Predictors and Encoders: Predictor training strategies were revamped with AdamW and cosine scheduling, switching the dataset encoder from a standard ResNet-18 to a Vision Transformer (ViT-B-16), substantially boosting rank correlation.
Datasets
- Meta-Datasets (Training): Built by randomly sampling 20-class subsets from ImageNet32 (1,000 samples per class) to train sampled architectures. The authors expanded the training cardinality to 10,000 triplets for NASBench201 and 20,000 triplets for MobileNetV3, tuning pipelines via Optuna hyperparameter optimization.
- Evaluation Benchmarks: Evaluated across 15 diverse image classification datasets, including CIFAR-10, CIFAR-100, FGVC Aircraft, Oxford III Pets, STL-10, TinyImageNet, EuroSAT, FashionMNIST, and several medical benchmarks from MedMNIST (BloodMNIST, DermaMNIST, OCTMNIST, OrganAMNIST, OrganCMNIST, PathMNIST, TissueMNIST).
- Search Spaces: Evaluated on cell-based NASBench201 (~15.6k architectures) and the macro-level MobileNetV3 search space implemented via Once-for-All (~$2 \times 10^{19}$ architectures).
Key Insights and Ideas
- Multi-Guide Gradient Steering Works in Graph Diffusion: Guiding diffusion with multiple independent neural score guides simultaneously does not collapse sample quality. Instead, balancing gradients allows fine control over competing physical constraints.
- Pareto Stretching Prevents Compromise Clustering: Standard multi-objective optimization often clusters solutions in a mediocre middle ground. Split-phase batch generation ("stretching") intentionally forces exploration toward opposite extremes of the trade-off surface.
- Zero-Shot Candidate Selection: Because denoised accuracy and efficiency metrics are reliably predicted upfront (Spearman's $\rho$ up to 0.88–0.98), the user only ever needs to train one network—the final chosen Pareto point—reducing downstream training costs to the absolute minimum.
Key Results
- Higher Accuracy with Fewer Trained Models: On NASBench201, $\text{POMONAG}_{\text{Acc}}$ sets new state-of-the-art accuracy benchmarks on CIFAR-10 (95.42%), CIFAR-100 (75.94%), Aircraft (63.38%), and Oxford Pets (68.82%), outperforming DiffusionNAG (which required evaluating multiple candidates) while training only 1 architecture per task.
- Comprehensive 15-Dataset Gains:
- NASBench201: Compared to DiffusionNAG (77.83% avg accuracy), $\text{POMONAG}_{\text{Acc}}$ achieved 81.89% average accuracy while still reducing parameters (0.65M vs. 1.02M), MACs (11.40M vs. 18.02M), and latency (13.68 ms vs. 18.83 ms).
- MobileNetV3: All POMONAG variants beat DiffusionNAG’s average accuracy (87.29%). $\text{POMONAG}_{\text{Acc}}$ reached 89.96% accuracy with a 24% reduction in parameters.
- Extreme Efficiency ($\text{POMONAG}_{\text{Eff}}$): On NASBench201, slashed parameter count by 90%, MACs by 93%, and latency by 65% relative to DiffusionNAG, giving edge practitioners ultra-compact options.
Limitations and Future Work
- Structural Validity Drop in Large Spaces: In the MobileNetV3 search space, validity dropped to 72.58% (compared to DiffusionNAG’s 99.09%), indicating that aggressively steering toward multiple competing bounds occasionally generates invalid architectural configurations.
- Task Scope: Experiments are currently limited to 2D image classification. The authors identify extending POMONAG to dense vision tasks (such as semantic segmentation and object detection) as an essential future direction to build a truly universal, task-aware NAS foundation model.
Summary updated