Two-stage architectural fine-tuning with neural architecture search using early-stopping in image classification

Youngkee Kim, Won Joon Yun, Youn Kyu Lee, Soyi Jung, Joongheon KimarXiv 2022 · arXiv

TL;DR

This paper proposes a two-stage approach to transfer learning that fine-tunes both a neural network's architecture and its weights. By combining Neural Architecture Search (NAS) with clever mutation rules and an early-stopping mechanism, the authors achieve higher accuracy on target tasks while reducing computational costs by 32.4% and search time by 22.3%.

What Problem Does This Paper Address?

Deep Learning thrives on massive datasets, but in many industrial applications, high-quality labeled data is scarce. Transfer learning—taking a model pre-trained on a giant dataset like ImageNet and "fine-tuning" it for a smaller, specific task—is the standard solution.

However, traditional fine-tuning is limited: it only updates the model's weights (the internal parameters), leaving the architecture (the arrangement of layers and operations) untouched. While Neural Architecture Search (NAS) can design custom architectures from scratch, it is notoriously "data-hungry" and computationally expensive, often requiring thousands of GPU hours. This paper aims to make architectural adaptation practical by creating a lightweight, two-stage fine-tuning process.

Methods

The authors introduce a Two-stage Architectural Fine-Tuning framework:

  1. Stage 1: Architectural Fine-Tuning: Instead of searching for a whole new network, they start with a validated base model (like ResNet-18). They define "mutable blocks" where specific operations can be changed—for example, swapping a 3x3 convolutional kernel for a 5x5 one to change the receptive field. An RL-based controller (using an LSTM) samples these "mutations" to find the best architecture for the new task.
  2. Stage 2: Weight Fine-Tuning: Once the optimal architecture is found, the model undergoes standard weight fine-tuning on the target dataset.
Figure 1. Overview of the proposed two-stage architectural fine-tuning process.

Key Innovations:

  • Mutation Rules & Search Scope: To keep the search space manageable, the authors only allow mutations in deeper layers, keeping shallow layers (which capture general features) frozen. They define four "scopes" (Small to Full) to control how much of the network is open to change.
  • Early-Stopping: The authors observed that the RL controller often settles on a specific architecture long before its internal rewards fully converge. By stopping the search once the "action distribution" (the choice of architecture) stabilizes, they significantly cut down on search time.
Figure 2. Search space definition.

Datasets

The researchers evaluated their method using a classic transfer learning setup:

  • Source Dataset: ImageNet, used for initial pre-training of the base models (ResNet-18 and ResNet-50).
  • Target Dataset: CIFAR10, representing the smaller, task-specific data where the model is fine-tuned.

Key Insights and Ideas

  • Architecture Matters in Transfer: The results confirm that just tuning weights isn't enough. Adapting the architecture—specifically the kernel sizes—allows the model to better capture the spatial features of the new dataset.
  • The "Action Stability" Heuristic: The most practical takeaway is the early-stopping method. In NAS, we usually wait for accuracy to peak. This paper shows that watching the consistency of the controller's choices is a much faster way to know when you've found a "good enough" design.
  • The Search Scope Trade-off: There is a delicate balance between flexibility and prior knowledge. A "Small" search scope (mutating only the very last layers) often performed better than a "Full" scope because it preserved more of the high-quality features learned from the massive ImageNet dataset.

Key Results

The proposed ENASResNet models consistently outperformed "Vanilla" fine-tuned models across various configurations:

  • Efficiency: The method reduced overall computational costs by 32.4% and search costs by 22.3%.
  • Speed to Accuracy: For ResNet-18, the ENAS version reached 85% test accuracy in just 46,000 iterations, whereas the vanilla model required 68,000 iterations to reach the same level.
  • Final Performance: In the "Small" scope configuration, ENASResNet-18 achieved a final accuracy of 87.4%, surpassing the vanilla model's 86.4%. Similar gains were seen with ResNet-50, with accuracy improvements of up to 4.0%.
Figure 3. Test accuracy results in the weight fine-tuning stage.

Limitations and Future Work

While the paper demonstrates clear gains, a few areas remain for exploration:

  • Task Diversity: The study focuses on image classification. It remains to be seen how these mutation rules perform on more complex tasks like object detection or semantic segmentation where architectural requirements are more stringent.
  • Mutation Variety: The current mutation rule is simple (kernel size swapping). Future work could incorporate more diverse operations like dilated convolutions or different activation functions into the mutable blocks.
  • Base Model Dependency: The method relies on having a strong base architecture. Its effectiveness on less-optimized "starting" models hasn't been tested.

Summary updated

Abstract

In many deep neural network (DNN) applications, the difficulty of gathering high-quality data in the industry field hinders the practical use of DNN. Thus, the concept of transfer learning has emerged, which leverages the pretrained knowledge of DNNs trained on large-scale datasets. Therefore, this paper suggests two-stage architectural fine-tuning, inspired by neural architecture search (NAS). One of main ideas is mutation, which reduces the search cost using given architectural information. Moreover, early-stopping is considered which cuts NAS costs by terminating the search process in advance. Experimental results verify our proposed method reduces 32.4% computational and 22.3% searching costs.
BibTeX
@misc{kim2022two,
  author = {Youngkee Kim and Won Joon Yun and Youn Kyu Lee and Soyi Jung and Joongheon Kim},
  title = {{Two-stage architectural fine-tuning with neural architecture search using early-stopping in image classification}},
  year = {2022},
  eprint = {2202.08604},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.08604},
}
Figure