Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models

Dongkuan Xu, Subhabrata Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Hassan Awadallah, Jianfeng GaoNeurIPS 2022 · OpenReview

TL;DR

AutoDistil is a novel framework that uses "few-shot" task-agnostic Neural Architecture Search (NAS) to distill large language models into a variety of smaller, efficient student models. By partitioning the search space into independent sub-spaces, it avoids the performance-sapping interference common in traditional NAS, delivering state-of-the-art compressed models with up to a 41x reduction in computational cost.

What Problem Does This Paper Address?

Large Language Models (LLMs) like BERT and GPT-3 are incredibly powerful but prohibitively expensive to run on edge devices or under tight latency constraints. While Knowledge Distillation (KD)—training a small "student" model to mimic a large "teacher"—is a popular solution, it has two major drawbacks:

  1. Manual Design: Engineers must manually guess the best student architecture (number of layers, heads, etc.) for a specific hardware budget, often requiring many trial-and-error runs.
  2. One-Shot Interference: Existing automated methods (NAS) usually try to train one giant "SuperLM" containing millions of possible sub-networks. This leads to "co-adaptation" or interference, where the training needs of a tiny sub-network conflict with those of a larger one, resulting in sub-optimal performance for both.

AutoDistil aims to automate this process while ensuring the resulting students are "task-agnostic"—meaning they can be fine-tuned for any downstream task (like sentiment analysis or question answering) without needing to be re-distilled.

Methods

The authors introduce three key innovations to overcome the limitations of traditional NAS and KD:

AutoDistil Framework Overview
  1. Few-shot Search Space Partitioning: Instead of one massive SuperLM, AutoDistil partitions the Transformer search space into $K$ compact sub-spaces (e.g., $K=3$ for "Base," "Small," and "Tiny" ranges). This "few-shot" approach allows each SuperLM to specialize, significantly reducing interference between sub-networks of vastly different sizes.
  2. Task-agnostic SuperLM Training: The SuperLMs are trained using a single-stage objective: Deep Self-Attention Distillation. They align the "internal logic" (the relationships between keys, queries, and values in the attention layers) of the student with the teacher. Crucially, this process uses unlabeled data and no task-specific labels.
  3. Lightweight Search: Once the $K$ SuperLMs are trained, finding the optimal student for a specific budget is "free." You simply evaluate the pre-trained sub-networks on a held-out validation set and pick the one with the lowest distillation loss. No additional training or "re-distillation" is required.

Datasets

  • Training/Distillation: The models were distilled using the English Wikipedia and BookCorpus datasets (standard for pre-training BERT-style models).
  • Evaluation: The framework was tested on the GLUE (General Language Understanding Evaluation) benchmark, which includes tasks like MNLI (inference), SST-2 (sentiment), MRPC (paraphrase), and CoLA (linguistic acceptability).

Key Insights and Ideas

  • The Power of Partitioning: The most significant "aha moment" is that splitting the search space into a few independent chunks (few-shot) performs much better than trying to cram everything into one model (one-shot). It provides enough capacity for the weights to adapt to specific size ranges.
  • Inductive Bias Matters: The authors incorporate heuristics—like the observation that "thinner and deeper" networks often perform better than "wider and shallower" ones—to prune the search space before training even begins.
  • Single-Stage Efficiency: Unlike many NAS methods that require a "pre-training -> search -> further distillation" pipeline, AutoDistil achieves competitive results in a single training stage, drastically reducing the total GPU hours required.

Key Results

Accuracy vs. FLOPs Trade-off
  • Superior Performance: AutoDistil outperformed state-of-the-art KD methods (like TinyBERT and MiniLM) and NAS methods (like DynaBERT) across the GLUE benchmark, establishing a clear Pareto frontier.
  • Massive Compression: The most aggressive student model achieved a 41x reduction in FLOPs compared to the BERT-Base teacher while maintaining respectable accuracy.
  • Cost Savings: AutoDistil reduced the computational cost of the NAS process by up to 41x compared to multi-stage NAS baselines. For example, the search phase takes less than one GPU hour, whereas competing methods often require hundreds.
  • Agnostic vs. Proxy: The "Task-agnostic" search (picking a student based on distillation loss) performed nearly as well as "Task-proxy" search (picking a student based on actual task performance), proving that the distillation loss is a high-quality signal for model quality.

Limitations and Future Work

  • Sub-network Extraction: The current method uses a simple "bottom-left" truncation to extract sub-networks from the SuperLM. The authors suggest that more sophisticated strategies, such as ranking attention heads by importance, could yield even better results.
  • Teacher Scaling: The experiments focused on BERT-Base as the teacher. Extending this framework to even larger models (like RoBERTa-Large or GPT-style decoders) remains an area for future exploration.
  • Heuristic Dependency: The partitioning into $K=3$ sub-spaces is a heuristic. Future work could investigate automated ways to determine the optimal number and boundaries of these partitions.

Summary updated

Abstract

Traditional knowledge distillation (KD) methods manually design student architectures to compress large models given pre-specified computational cost. This requires several trials to find viable students, and repeating the process with change in computational budget. We use Neural Architecture Search (NAS) to automatically distill several compressed students with variable cost from a large model. Existing NAS methods train a single SuperLM consisting of millions of subnetworks with weight-sharing, resulting in interference between subnetworks of different sizes. Additionally, many of these works are task-specific requiring task labels for SuperLM training. Our framework AutoDistil addresses above challenges with the following steps: (a) Incorporates inductive bias and heuristics to partition Transformer search space into K compact sub-spaces (e.g., K=3 can generate typical student sizes of base, small and tiny); (b) Trains one SuperLM for each sub-space using task-agnostic objective (e.g., self-attention distillation) with weight-sharing of students; (c) Lightweight search for the optimal student without re-training. Task-agnostic training and search allow students to be reused for fine-tuning on any downstream task. Experiments on GLUE benchmark demonstrate AutoDistil to outperform state-of-the-art KD and NAS methods with upto 3x reduction in computational cost and negligible loss in task performance. Code and model checkpoints are available at https://github.com/microsoft/autodistil.
BibTeX
@inproceedings{xu2022few,
  author = {Dongkuan Xu and Subhabrata Mukherjee and Xiaodong Liu and Debadeepta Dey and Wenhui Wang and Xiang Zhang and Ahmed Hassan Awadallah and Jianfeng Gao},
  title = {{Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=kgvFxxXjco},
}
Figure