TL;DR
This paper introduces AutoReCon, a novel method that uses Neural Architecture Search (NAS) to automatically design the generator architecture for data-free compression. By optimizing the generator's structure, AutoReCon significantly improves the performance of data-free compression techniques, outperforming existing methods that rely on human-designed generators.
What Problem Does This Paper Address?
Deep learning models are increasingly deployed on resource-constrained hardware, necessitating efficient model compression techniques like quantization and knowledge distillation. However, many of these compression methods require the original training dataset. When this dataset is unavailable due to privacy concerns or transmission limitations (data-free compression), a crucial step is to reconstruct a synthetic training dataset. Current approaches reconstruct this dataset using a generator, which is typically designed by humans. These methods focus on extracting more information from the pre-trained model to improve reconstruction quality. This paper identifies a gap: the potential benefits of optimizing the generator's architecture itself, rather than just its weights, for data-free compression have been overlooked. AutoReCon aims to fill this gap by automating the design of the generator architecture.
Methods
The core of AutoReCon is its application of Neural Architecture Search (NAS) to design the generator. Instead of relying on human intuition, AutoReCon searches for an optimal generator architecture.
- Stochastic Super Net: The method defines a flexible "super net" that encompasses a wide range of possible generator architectures. This super net is parameterized by architecture parameters.
- Layer-wise Search Space: The search space is defined at a layer-wise level, specifying different types of connections (normal-edge, up-edge, cross-edge) and offering a mixture of candidate operations (e.g., different convolution types, interpolation) for each edge.
- Gumbel Softmax: To make the discrete architecture search differentiable, the paper employs the Gumbel Softmax trick to obtain continuous approximations of sampling probabilities for operations.
- Two-Stage Training:
- Stage 1 (Architecture Search): The super net is trained to minimize the reconstruction loss ($L_r$). This stage involves two phases: updating the weights of the super net to improve reconstruction quality and updating the architecture parameters to discover better architectures.
- Stage 2 (Compression): Once an optimized generator architecture is discovered, it's used to generate the training data, and then the target model is compressed (e.g., quantized) using this data to minimize a compression loss ($L_c$).
Datasets
The paper evaluates its method on two standard image classification datasets:
- CIFAR-100: Used for training and evaluation.
- ImageNet: Used for evaluation.
The authors compare their results against baselines that also use these datasets. No new benchmarks were introduced.
Key Insights and Ideas
- Network Engineering Matters for Reconstruction: The paper's central insight is that the architecture of the generator used for data reconstruction is as important, if not more so, than simply extracting more information from the pre-trained model. Optimizing the generator's structure can lead to better reconstructed data.
- NAS for Generator Design is Effective: Applying NAS to design the generator architecture for data-free compression yields significant performance improvements, demonstrating that automated architecture search can outperform human-designed architectures in this specific task.
- Scalability of Discovered Architectures: The AutoReCon-discovered generator architectures exhibit superior scalability compared to human-designed ones. When scaling base channels, the AutoReCon generators consistently improve performance, while human-designed ones may degrade.
- Complementary Approaches: The paper highlights that network engineering (NAS) and exploiting prior information from the pre-trained model are complementary, not contradictory, and both are crucial for advancing data-free compression.
Key Results
AutoReCon consistently improves the performance of data-free compression methods across various pre-trained models and quantization levels.
- ImageNet: Using ResNet18 and 3-bit quantization, AutoReCon improved the Top-1 accuracy of the GDFQ method by 2.68%. With MobileNetV2 and 3-bit quantization, the improvement was 11.99%. For ResNet50 and 5-bit quantization, AutoReCon surpassed GDFQ by 8.43%.
- CIFAR-100: With ResNet18 and 3-bit quantization, AutoReCon improved the Top-1 accuracy by 17.73%. For MobileNetV2 and 5-bit quantization, the improvement was 4.51%. With ResNet50 and 4-bit quantization, the improvement was 9.66%.
- The discovered generator architectures require almost the same parameters and fewer flops compared to human-designed counterparts.
Limitations and Future Work
- Computational Cost of NAS: While effective, NAS methods can be computationally expensive. The paper mentions techniques like weight sharing and progressive search to reduce this cost, but it remains a factor.
- Generalization to Other Tasks: The current work focuses on image classification. Future work could explore the applicability of AutoReCon to other domains and tasks.
- Exploring Different Search Spaces: The paper defines a specific layer-wise search space. Investigating alternative search spaces or NAS algorithms could yield further improvements.
- Integration with Other Compression Techniques: While AutoReCon is shown to work with GDFQ and GFAD, exploring its integration with a wider range of data-free compression techniques is a natural next step.
Summary updated