Graph Masked Autoencoder Enhanced Predictor for Neural Architecture Search

Kun Jing, Jungang Xu, Pengfei LiIJCAI 2022 · DOI

TL;DR

This paper introduces a Graph Masked Autoencoder (GMAE) to pre-train neural architecture performance predictors. By leveraging self-supervised learning on a vast pool of unlabeled architectures, GMAE significantly reduces the reliance on expensive, labeled architecture evaluations, leading to more efficient and effective Neural Architecture Search (NAS).

What Problem Does This Paper Address?

Neural Architecture Search (NAS) aims to automate the design of optimal neural networks for specific tasks. A crucial component of many NAS methods is a "performance predictor," which estimates an architecture's performance without requiring a full, time-consuming training and evaluation cycle. However, training these predictors effectively typically requires a substantial amount of labeled data (i.e., architectures that have already been fully trained and evaluated). This creates a bottleneck, as obtaining this labeled data is computationally expensive. Existing methods struggle to balance the cost of data acquisition with predictor accuracy, especially when dealing with massive search spaces containing millions of potential architectures. This paper addresses the challenge of training accurate performance predictors with limited labeled data by proposing a self-supervised pre-training approach.

Methods

The core innovation is the Graph Masked Autoencoder (GMAE). GMAE treats neural architectures as directed acyclic graphs (DAGs) and employs a self-supervised learning strategy inspired by masked language modeling (like BERT) and masked autoencoding (like MAE).

Here's a breakdown:

  • Architecture Representation: Architectures are represented as DAGs, where vertices correspond to operations (e.g., convolutions, pooling) and edges represent data flow. This DAG is further encoded using an adjacency matrix and a vertex-type matrix.
  • Encoder: A Graph Attention Network (GAT) acts as the encoder, processing the DAG representation to learn embeddings for each vertex.
  • Masking: During pre-training, a random subset of vertices (operations) in the DAG is "masked."
  • Decoder: A simple linear projection followed by a SoftMax function acts as the decoder. Its task is to reconstruct the original operations at the masked vertices.
  • Pre-training Objective: The model is trained to minimize the cross-entropy error between the predicted operations and the actual operations at the masked vertices. This generative approach allows the model to learn the underlying structure and relationships within architectures without explicit performance labels.
  • Fine-tuning: After pre-training, the encoder's learned representations are used. The decoder is discarded, and a new predictor (a regressor) is added. This combined model is then fine-tuned using a small amount of labeled data (actual architecture performance scores). The paper explores both end-to-end fine-tuning and partial fine-tuning (freezing encoder weights).
  • Search Enhancement: The fine-tuned predictor is then integrated with various NAS search strategies (Random Search, Reinforcement Learning, Aging Evolution, Bayesian Optimization) to improve their query utilization and discover better architectures.

Datasets

The paper evaluates its method on two well-established NAS benchmarks:

  • NAS-Bench-101: A search space of 423K unique convolutional architectures, providing validation and test accuracies on CIFAR-10.
  • NAS-Bench-301 (DARTS): A larger search space of approximately 10^18 architectures, also evaluated on CIFAR-10. The authors adapt this operation-on-edge space to their operation-on-vertex representation.

The core idea is to leverage the vast number of untrained architectures within these search spaces as unlabeled data for GMAE pre-training.

Key Insights and Ideas

  • Self-Supervised Pre-training for NAS Predictors: The most significant insight is that self-supervised learning, particularly generative masked reconstruction, can effectively pre-train architecture encoders. This pre-training captures structural information about architectures, making them more amenable to learning performance prediction with far less labeled data.
  • Reduced Dependence on Labeled Data: GMAE dramatically lowers the barrier to entry for predictor-based NAS by minimizing the need for expensive, fully trained architectures. This is crucial for exploring massive search spaces.
  • Generative vs. Discriminative Self-Supervision: The paper finds that generative self-supervised learning (masked reconstruction) is more effective for architecture performance prediction than discriminative methods like graph contrastive learning.
  • Masking Ratio Sensitivity: The optimal masking ratio for GMAE pre-training is shown to be sensitive to the search space size and characteristics.
  • BPR Loss for Fine-tuning: Bayesian Personalized Ranking (BPR) loss is identified as a superior objective for fine-tuning predictors, as it focuses on relative ranking rather than absolute performance, leading to more stable and accurate predictions.

Key Results

  • Improved Predictor Performance: GMAE-enhanced predictors achieve significantly higher Kendall's τ (a measure of rank correlation) compared to SOTA predictors, especially when training data is scarce (e.g., less than 1500 samples). For instance, with fewer than 500 training samples in NAS-Bench-101, GMAE achieved a Kendall's τ about 0.1 higher than the best existing predictor.
  • Enhanced Search Strategies: When integrated with search algorithms, GMAE-NAS methods consistently outperform or match SOTA predictor-based NAS methods (BANANAS, CATE) in terms of test error and query utilization across both NAS-Bench-101 and NAS-Bench-301.
  • Competitive Real-World Performance: The architectures discovered by GMAE-NAS achieved competitive results on the actual NAS task (CIFAR-10), demonstrating practical applicability.
  • Efficiency: The pre-training process, while computationally intensive, is a one-time cost. The subsequent fine-tuning and search phases are significantly faster due to the reduced need for labeled data.

Limitations and Future Work

  • Computational Cost of Pre-training: While GMAE reduces the need for labeled data during fine-tuning and search, the initial GMAE pre-training itself can be computationally demanding.
  • Masking Ratio Sensitivity: The need to tune the masking ratio for different search spaces adds a hyperparameter tuning step.
  • Transformer vs. GNN: The paper notes that data-hungry Transformer models struggle with limited training data, favoring GNNs like GAT for this task. Future work could explore more efficient Transformer variants or hybrid approaches.
  • Generalizability: While GMAE shows promise across different search spaces, further investigation into its generalizability to even more diverse and complex NAS benchmarks would be valuable.
  • Predictor Quality in Large Spaces: The paper acknowledges that when a very large number of training samples are used for fine-tuning, GMAE's performance can be surpassed by other methods, suggesting room for improvement in convergence or fine-tuning strategies for extremely large datasets.

Summary updated

BibTeX
@inproceedings{jing2022graph,
  author = {Kun Jing and Jungang Xu and Pengfei Li},
  title = {{Graph Masked Autoencoder Enhanced Predictor for Neural Architecture Search}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/432},
}
Figure