NEAP-F: Network Epoch Accuracy Prediction Framework (Student Abstract)

Arushi Chauhan, Mayank Vatsa, Richa Singh

Introduction

Neural Architecture Search (NAS) has significantly advanced deep learning by automating the design of high-performing models. However, the search process is computationally expensive, often requiring thousands of candidate architectures to be fully trained to evaluate their performance. This consumes massive amounts of GPU time and energy. The Network Epoch Accuracy Prediction Framework (NEAP-F) addresses this bottleneck by predicting a convolutional neural network's (CNN) accuracy on a given dataset in milliseconds, without requiring the network to be trained.

Methodology

NEAP-F utilizes a two-pronged vectorization strategy to represent both the model architecture and the dataset characteristics, which are then fed into a regression model to predict performance.

1. Vectorizing Architecture

To account for modern architectures that utilize skip connections (e.g., ResNets), NEAP-F treats a neural network as a graph. Each connection is modeled as an edge between a source layer ($i$) and a destination layer ($j$). A weight is assigned to each edge based on the ratio of output to input channels:

$$\text{weight}(i, j) = \frac{\text{number of output channels of layer } i}{\text{number of input channels of layer } j}$$

This approach allows complex, non-linear topologies to be represented as standardized vectors, making skip connections first-class citizens in the prediction process.

2. Vectorizing Dataset "Ease"

A model's accuracy is intrinsically linked to the difficulty of the dataset. NEAP-F identifies "pivotal factors" for prediction, including the number of classes, class imbalance, and the ratio of inter-class to intra-class similarity. By quantifying these characteristics, the framework can generalize across different types of data.

3. The Predictor

The resulting vectors, along with the target epoch number, are processed by a regressor. The authors evaluated several models, including Ordinary Least Squares (OLS), SVM, XGBoost, and Random Forest, to map these inputs to a predicted accuracy value between 0 and 1.

Experimental Results

The framework was evaluated using 50,670 triplets of <sample network, image dataset, epoch> derived from the MetaQNN dataset and standard benchmarks (CIFAR-10, MNIST, and SVHN).

Model Performance

Among the tested regressors, the Random Forest Regressor consistently emerged as the top performer, achieving the lowest Mean Squared Error (MSE) and Mean Absolute Error (MAE) across the board.

Table 1: Performance comparison of various regression models showing Random Forest as the most accurate predictor.

Data Efficiency

A key finding was the framework's data efficiency. Even when the regression models were trained on smaller subsets of the architecture data, they showed rapid improvements in prediction accuracy. This suggests that the vectorization method effectively captures the essential features of the networks.

Figure 1: Testing Mean Squared Error vs. Training Set Size, illustrating how prediction accuracy improves as more architecture samples are provided.

On specific datasets, the Random Forest model achieved impressively low MSE values: 0.0023 for CIFAR-10, 0.0230 for MNIST, and 0.0032 for SVHN.

Conclusion

NEAP-F demonstrates that it is possible to "hallucinate" the performance of thousands of models in milliseconds by treating dataset difficulty and architecture topology as quantifiable inputs. While future work is needed to scale this to massive datasets like ImageNet and incorporate more complex layer types (such as attention mechanisms), NEAP-F provides a promising path toward more sustainable and efficient Neural Architecture Search.

Summary updated

Abstract

Recent work in neural architecture search has spawned interest in algorithms that can predict the performance of convolutional neural networks using minimum time and computation resources. We propose a new framework, Network Epoch Accuracy Prediction Framework (NEAP-F) which can predict the testing accuracy achieved by a convolutional neural network in one or more epochs. We introduce a novel approach to generate vector representations for networks, and encode ``ease" of classifying image datasets into a vector. For vector representations of networks, we focus on the layer parameters and connections between the network layers. A network achieves different accuracies on different image datasets; therefore, we use the image dataset characteristics to create a vector signifying the ``ease" of classifying the image dataset. After generating these vectors, the prediction models are trained with architectures having skip connections seen in current state-of-the-art architectures. The framework predicts accuracies in order of milliseconds, demonstrating its computational efficiency. It can be easily applied to neural architecture search methods to predict the performance of candidate networks and can work on unseen datasets as well.
BibTeX
@inproceedings{chauhan2021neap,
  author = {Arushi Chauhan and Mayank Vatsa and Richa Singh},
  title = {{NEAP-F: Network Epoch Accuracy Prediction Framework (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17880},
}
Figure