Sure, here's a blog-style summary of the paper "Multi-Fidelity Neural Architecture Search With Knowledge Distillation":
TL;DR
This paper introduces MF-KD, a novel method for neural architecture search (NAS) that significantly speeds up the process by using knowledge distillation (KD) to improve low-fidelity evaluations. By training networks for fewer epochs with a KD-enhanced loss function, MF-KD achieves better architecture selection and outperforms existing methods on standard benchmarks.
What Problem Does This Paper Address?
Neural Architecture Search (NAS) aims to automate the discovery of optimal neural network architectures. However, evaluating each candidate architecture by training it fully is computationally expensive and time-consuming. Existing methods try to mitigate this by using "low-fidelity" evaluations (e.g., training for fewer epochs or on a subset of data). The challenge is that these low-fidelity evaluations can be biased and may not accurately reflect the true performance of an architecture, leading to suboptimal choices. This paper addresses the problem of improving the reliability of low-fidelity evaluations in NAS to enable faster and more effective architecture discovery.
Methods
The core innovation of this paper is the Multi-Fidelity Neural Architecture Search with Knowledge Distillation (MF-KD) method. It combines two key ideas:
- Knowledge Distillation (KD) for Low-Fidelity Evaluations: Instead of using a standard logistic loss for low-fidelity training (e.g., few epochs), MF-KD employs a knowledge distillation loss. This loss encourages the "student" network (the architecture being evaluated) to mimic the outputs of a pre-trained "teacher" network. The authors hypothesize that this "dark knowledge" captured by KD helps the student network learn more robust representations even with limited training, thus improving the correlation between low- and high-fidelity evaluations. They specifically use the NST loss (Maximum Mean Discrepancy) as a form of KD.
- Bayesian Multi-Fidelity Optimization: MF-KD integrates these improved low-fidelity evaluations into a Bayesian optimization framework. Specifically, it uses a co-kriging fusion model to combine information from both low-fidelity (using KD) and high-fidelity (full training) evaluations. This allows the search to intelligently balance exploration (trying new architectures) and exploitation (focusing on promising ones) while leveraging the speed of low-fidelity assessments.
Datasets
The experiments were conducted on the following datasets:
- CIFAR-10
- CIFAR-100
- ImageNet-16-120 (a downsampled variant of ImageNet)
These datasets were used in conjunction with the NAS-Bench-201 benchmark, which provides a large, fixed search space of convolutional architectures.
Key Insights and Ideas
- KD as a "Better Teacher" for Low-Fidelity NAS: The most significant insight is that knowledge distillation, when applied during low-fidelity training, can significantly improve the correlation between low-fidelity and high-fidelity performance metrics. This means architectures that perform well under KD-enhanced low-fidelity training are more likely to be good architectures overall.
- Bridging the Gap Between Speed and Accuracy: MF-KD effectively bridges the gap between computationally expensive high-fidelity evaluations and potentially misleading low-fidelity ones. By making low-fidelity evaluations more informative, the method allows for more efficient exploration of the vast NAS search space.
- Structural Knowledge Transfer: The paper suggests that KD can transfer structural knowledge from a teacher architecture to the student, which is beneficial for architecture search.
Key Results
- Improved Architecture Selection: MF-KD consistently outperformed several state-of-the-art baselines, including DARTS, GDAS, ENAS, Regularized Evolution, Random Search, and BOHB, in terms of the accuracy of the best found architecture.
- Significant Performance Gains: For CIFAR-100 and ImageNet-16-120, MF-KD found architectures with accuracies of 72.00% and 45.67% respectively, often outperforming the next best method by a noticeable margin.
- Higher Correlation: Table 1 shows that training with the NST loss (a KD method) significantly improves the Kendall-tau correlation between low-fidelity (1 epoch) and high-fidelity evaluations compared to using a conventional logistic loss.
- Ablation Study Validation: Ablation studies confirmed that both the multi-fidelity search strategy and the KD-enhanced low-fidelity evaluations contribute to the overall performance improvement.
Limitations and Future Work
- Computational Overhead of KD: While MF-KD is faster than pure high-fidelity NAS, calculating the gradient for some KD losses (like NST) can add significant overhead compared to logistic loss. The authors explored an approximation for NST to mitigate this.
- Teacher Architecture Dependence: The paper notes that the found architecture can depend on the teacher architecture used for KD, implying that selecting an appropriate teacher is important.
- Extension to More Fidelity Levels: The proposed model can be extended to more than two levels of fidelity, which could further refine the search process.
- Layer Selection for KD: The paper suggests that selecting which layers to apply KD to is an interesting area for future research.
Summary updated