Multi-fidelity Neural Architecture Search with Knowledge Distillation

Ilya Trofimov, Nikita Klyuchnikov, Mikhail Salnikov, Alexander Filippov, Evgeny Burnaev

Sure, here's a blog-style summary of the paper "Multi-Fidelity Neural Architecture Search With Knowledge Distillation":

TL;DR

This paper introduces MF-KD, a novel method for neural architecture search (NAS) that significantly speeds up the process by using knowledge distillation (KD) to improve low-fidelity evaluations. By training networks for fewer epochs with a KD-enhanced loss function, MF-KD achieves better architecture selection and outperforms existing methods on standard benchmarks.

What Problem Does This Paper Address?

Neural Architecture Search (NAS) aims to automate the discovery of optimal neural network architectures. However, evaluating each candidate architecture by training it fully is computationally expensive and time-consuming. Existing methods try to mitigate this by using "low-fidelity" evaluations (e.g., training for fewer epochs or on a subset of data). The challenge is that these low-fidelity evaluations can be biased and may not accurately reflect the true performance of an architecture, leading to suboptimal choices. This paper addresses the problem of improving the reliability of low-fidelity evaluations in NAS to enable faster and more effective architecture discovery.

Methods

The core innovation of this paper is the Multi-Fidelity Neural Architecture Search with Knowledge Distillation (MF-KD) method. It combines two key ideas:

  • Knowledge Distillation (KD) for Low-Fidelity Evaluations: Instead of using a standard logistic loss for low-fidelity training (e.g., few epochs), MF-KD employs a knowledge distillation loss. This loss encourages the "student" network (the architecture being evaluated) to mimic the outputs of a pre-trained "teacher" network. The authors hypothesize that this "dark knowledge" captured by KD helps the student network learn more robust representations even with limited training, thus improving the correlation between low- and high-fidelity evaluations. They specifically use the NST loss (Maximum Mean Discrepancy) as a form of KD.
  • Bayesian Multi-Fidelity Optimization: MF-KD integrates these improved low-fidelity evaluations into a Bayesian optimization framework. Specifically, it uses a co-kriging fusion model to combine information from both low-fidelity (using KD) and high-fidelity (full training) evaluations. This allows the search to intelligently balance exploration (trying new architectures) and exploitation (focusing on promising ones) while leveraging the speed of low-fidelity assessments.

Datasets

The experiments were conducted on the following datasets:

  • CIFAR-10
  • CIFAR-100
  • ImageNet-16-120 (a downsampled variant of ImageNet)

These datasets were used in conjunction with the NAS-Bench-201 benchmark, which provides a large, fixed search space of convolutional architectures.

Key Insights and Ideas

  • KD as a "Better Teacher" for Low-Fidelity NAS: The most significant insight is that knowledge distillation, when applied during low-fidelity training, can significantly improve the correlation between low-fidelity and high-fidelity performance metrics. This means architectures that perform well under KD-enhanced low-fidelity training are more likely to be good architectures overall.
  • Bridging the Gap Between Speed and Accuracy: MF-KD effectively bridges the gap between computationally expensive high-fidelity evaluations and potentially misleading low-fidelity ones. By making low-fidelity evaluations more informative, the method allows for more efficient exploration of the vast NAS search space.
  • Structural Knowledge Transfer: The paper suggests that KD can transfer structural knowledge from a teacher architecture to the student, which is beneficial for architecture search.

Key Results

  • Improved Architecture Selection: MF-KD consistently outperformed several state-of-the-art baselines, including DARTS, GDAS, ENAS, Regularized Evolution, Random Search, and BOHB, in terms of the accuracy of the best found architecture.
  • Significant Performance Gains: For CIFAR-100 and ImageNet-16-120, MF-KD found architectures with accuracies of 72.00% and 45.67% respectively, often outperforming the next best method by a noticeable margin.
  • Higher Correlation: Table 1 shows that training with the NST loss (a KD method) significantly improves the Kendall-tau correlation between low-fidelity (1 epoch) and high-fidelity evaluations compared to using a conventional logistic loss.
  • Ablation Study Validation: Ablation studies confirmed that both the multi-fidelity search strategy and the KD-enhanced low-fidelity evaluations contribute to the overall performance improvement.

Limitations and Future Work

  • Computational Overhead of KD: While MF-KD is faster than pure high-fidelity NAS, calculating the gradient for some KD losses (like NST) can add significant overhead compared to logistic loss. The authors explored an approximation for NST to mitigate this.
  • Teacher Architecture Dependence: The paper notes that the found architecture can depend on the teacher architecture used for KD, implying that selecting an appropriate teacher is important.
  • Extension to More Fidelity Levels: The proposed model can be extended to more than two levels of fidelity, which could further refine the search process.
  • Layer Selection for KD: The paper suggests that selecting which layers to apply KD to is an interesting area for future research.

Summary updated

Abstract

Neural architecture search (NAS) targets at finding the optimal architecture of a neural network for a problem or a family of problems. Evaluations of neural architectures are very time-consuming. One of the possible ways to mitigate this issue is to use low-fidelity evaluations, namely training on a part of a dataset, fewer epochs, with fewer channels, etc. In this paper, we propose a bayesian multi-fidelity method for neural architecture search: MF-KD. The method relies on a new approach to low-fidelity evaluations of neural architectures by training for a few epochs using a knowledge distillation. Knowledge distillation adds to a loss function a term forcing a network to mimic some teacher network. We carry out experiments on CIFAR-10, CIFAR-100, and ImageNet-16-120. We show that training for a few epochs with such a modified loss function leads to a better selection of neural architectures than training for a few epochs with a logistic loss. The proposed method outperforms several state-of-the-art baselines.
BibTeX
@inproceedings{trofimov2020multi,
  author = {Ilya Trofimov and Nikita Klyuchnikov and Mikhail Salnikov and Alexander Filippov and Evgeny Burnaev},
  title = {{Multi-fidelity Neural Architecture Search with Knowledge Distillation}},
  booktitle = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2023.3234810},
}
Figure