Introduction
Neural Architecture Search (NAS) has significantly advanced deep learning by automating the design of high-performing models. However, the search process is computationally expensive, often requiring thousands of candidate architectures to be fully trained to evaluate their performance. This consumes massive amounts of GPU time and energy. The Network Epoch Accuracy Prediction Framework (NEAP-F) addresses this bottleneck by predicting a convolutional neural network's (CNN) accuracy on a given dataset in milliseconds, without requiring the network to be trained.
Methodology
NEAP-F utilizes a two-pronged vectorization strategy to represent both the model architecture and the dataset characteristics, which are then fed into a regression model to predict performance.
1. Vectorizing Architecture
To account for modern architectures that utilize skip connections (e.g., ResNets), NEAP-F treats a neural network as a graph. Each connection is modeled as an edge between a source layer ($i$) and a destination layer ($j$). A weight is assigned to each edge based on the ratio of output to input channels:
$$\text{weight}(i, j) = \frac{\text{number of output channels of layer } i}{\text{number of input channels of layer } j}$$
This approach allows complex, non-linear topologies to be represented as standardized vectors, making skip connections first-class citizens in the prediction process.
2. Vectorizing Dataset "Ease"
A model's accuracy is intrinsically linked to the difficulty of the dataset. NEAP-F identifies "pivotal factors" for prediction, including the number of classes, class imbalance, and the ratio of inter-class to intra-class similarity. By quantifying these characteristics, the framework can generalize across different types of data.
3. The Predictor
The resulting vectors, along with the target epoch number, are processed by a regressor. The authors evaluated several models, including Ordinary Least Squares (OLS), SVM, XGBoost, and Random Forest, to map these inputs to a predicted accuracy value between 0 and 1.
Experimental Results
The framework was evaluated using 50,670 triplets of <sample network, image dataset, epoch> derived from the MetaQNN dataset and standard benchmarks (CIFAR-10, MNIST, and SVHN).
Model Performance
Among the tested regressors, the Random Forest Regressor consistently emerged as the top performer, achieving the lowest Mean Squared Error (MSE) and Mean Absolute Error (MAE) across the board.
Data Efficiency
A key finding was the framework's data efficiency. Even when the regression models were trained on smaller subsets of the architecture data, they showed rapid improvements in prediction accuracy. This suggests that the vectorization method effectively captures the essential features of the networks.
On specific datasets, the Random Forest model achieved impressively low MSE values: 0.0023 for CIFAR-10, 0.0230 for MNIST, and 0.0032 for SVHN.
Conclusion
NEAP-F demonstrates that it is possible to "hallucinate" the performance of thousands of models in milliseconds by treating dataset difficulty and architecture topology as quantifiable inputs. While future work is needed to scale this to massive datasets like ImageNet and incorporate more complex layer types (such as attention mechanisms), NEAP-F provides a promising path toward more sustainable and efficient Neural Architecture Search.
Summary updated