You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient

Shaokun Zhang, Xiawu Zheng, Chenyi Yang, Yuchao Li, Yan Wang, Fei Chao, Mengdi Wang, Shen Li, Jun Yang, Rongrong Ji

TL;DR

YOCO-BERT ("You Only Compress Once") introduces an elastic compression framework that trains a single "super-BERT" model capable of instantly adapting to diverse hardware constraints without any post-search fine-tuning. By leveraging a novel Exploit-Explore Stochastic Natural Gradient (EE-SNG) mathematical framework, it navigates a massive search space of $10^{13}$ architectures to deliver state-of-the-art accuracy across various device budgets with an $O(1)$ training complexity for $N$ different devices.

What Problem Does This Paper Address?

Deploying large pre-trained language models like BERT onto resource-constrained edge devices (smartphones, IoT devices, etc.) is notoriously difficult due to their massive memory footprints and computational demands. While model compression techniques like pruning, quantization, and knowledge distillation help, they suffer from a major scalability flaw: they are rigid.

If you want to deploy a model to a device with a 50MB memory limit, you must compress and fine-tune it specifically for that budget. If your next target device only allows a 30MB budget, you have to repeat the entire expensive compression and fine-tuning process from scratch. Existing "elastic" compression paradigms attempt to solve this by training a single network containing multiple sub-networks, but when applied to BERT, they either restrict the search space too much or get trapped in local minima during the architecture search.

Methods

YOCO-BERT decouples the compression process into two distinct, sequential stages:

Figure 1: The proposed YOCO-BERT. We separate YOCO-BERT into two stages. Given expanded training space, we first propose a EE-SNG to optimize the probability distribution Pθ and super-BERT. In the second stage, searching algorithm is supervised by the optimized Pθ to obtain the optimal architecture across diverse constraints.
  1. Stage I: Joint Optimization (Super-BERT Training) The authors define a massive search space encompassing $10^{13}$ possible configurations by making three core dimensions elastic: the number of encoder layers, the intermediate size of the feed-forward networks (FFN), and the number of attention heads.

To train this parameter-sharing "super-BERT," they introduce the Exploit-Explore Stochastic Natural Gradient (EE-SNG) algorithm. Instead of randomly sampling sub-networks, EE-SNG treats Neural Architecture Search (NAS) as a continuous, differentiable likelihood maximization process. It maintains two probability distributions: an exploitation distribution (which optimizes towards high-performing architectures using natural gradients) and an exploration distribution (a uniform distribution over the entire search space). An online Dynamic Controller uses the information entropy of the exploitation distribution to dynamically balance the two, preventing the search from getting stuck in local minima.

  1. Stage II: Hardware Search Once the super-BERT is trained, its weights are frozen. For any given hardware constraint (e.g., a specific FLOP or parameter limit), a lightweight search is performed using a constraint-sensitive reward function. The optimal sub-network directly inherits its weights from the super-BERT—requiring zero fine-tuning.

Datasets

The authors evaluated YOCO-BERT on the widely accepted GLUE (General Language Understanding Evaluation) benchmark, testing across various tasks including:

  • SST-2 (Sentiment Analysis)
  • MRPC, QQP (Paraphrase & Semantic Equivalence)
  • CoLA (Linguistic Acceptability)
  • RTE, MNLI, QNLI (Natural Language Inference)

Note: Following standard practices, the controversial WNLI dataset was excluded.

Key Insights and Ideas

  • True Elasticity ($O(1)$ Complexity): Because the super-model is optimized globally, searching for $N$ different device configurations takes virtually no additional training time. The training complexity remains $O(1)$ relative to the number of target devices.
  • The Fine-Tuning Paradox: Surprisingly, the authors found that attempting to fine-tune the sub-networks after extracting them from the super-BERT actually degraded or failed to improve performance. This indicates that the joint optimization phase trains the shared weights so effectively that post-hoc adjustments disrupt their delicate balance.
  • Entropy-Driven Exploration: Using information entropy to control the explore/exploit boundary proved vital. When the model starts converging too quickly around a local architecture type (low entropy), the dynamic controller forces exploration across the broader uniform distribution to discover globally superior structures.

Key Results

YOCO-BERT consistently outperformed existing state-of-the-art compression methods (such as DistilBERT, DynaBERT, and BERT-of-Theseus) across the board.

Figure 2: Comparison of parameters, FLOPs between YOCO-BERT and previous works under different datasets. We use accuracy as the metric for the three datasets. Since YOCO-BERT does not use external data, Dyanbert’s results are obtained without using data augmentation (DA).
  • GLUE Benchmark Superiority: YOCO-BERT achieved a 2.1% to 4.5% average accuracy improvement over competing algorithms on the GLUE benchmark.
  • Tight Constraints, High Performance: Under a strict parameter budget of ~67M parameters, YOCO-BERT achieved an average GLUE score of 82.3%, outperforming the original 110M parameter $\text{BERT}_{\text{base}}$ baseline (81.7%) despite being nearly half the size.
  • Task-Specific Wins: On the CoLA dataset, YOCO-BERT achieved a massive 14.4% accuracy increase compared to LayerDrop at the same parameter scale.

Limitations and Future Work

While highly effective for BERT and encoder-only architectures on text classification tasks, the paper does not explore how this elastic framework scales to autoregressive generative models (like GPT-style decoders) or vision transformers (ViTs). The authors note that future work will focus on extending this "compress once, deploy everywhere" methodology to broader model families and a wider variety of complex NLP tasks.

Summary updated

Abstract

Despite superior performance on various natural language processing tasks, pre-trained models such as BERT are challenged by deploying on resource-constraint devices. Most existing model compression approaches require re-compression or fine-tuning across diverse constraints to accommodate various hardware deployments. This practically limits the further application of model compression. Moreover, the ineffective training and searching process of existing elastic compression paradigms[4,27] prevents the direct migration to BERT compression. Motivated by the necessity of efficient inference across various constraints on BERT, we propose a novel approach, YOCO-BERT, to achieve compress once and deploy everywhere. Specifically, we first construct a huge search space with 10^13 architectures, which covers nearly all configurations in BERT model. Then, we propose a novel stochastic nature gradient optimization method to guide the generation of optimal candidate architecture which could keep a balanced trade-off between explorations and exploitation. When a certain resource constraint is given, a lightweight distribution optimization approach is utilized to obtain the optimal network for target deployment without fine-tuning. Compared with state-of-the-art algorithms, YOCO-BERT provides more compact models, yet achieving 2.1%-4.5% average accuracy improvement on the GLUE benchmark. Besides, YOCO-BERT is also more effective, e.g.,the training complexity is O(1)for N different devices. Code is availablehttps://github.com/MAC-AutoML/YOCO-BERT.
BibTeX
@misc{zhang2021you,
  author = {Shaokun Zhang and Xiawu Zheng and Chenyi Yang and Yuchao Li and Yan Wang and Fei Chao and Mengdi Wang and Shen Li and Jun Yang and Rongrong Ji},
  title = {{You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient}},
  year = {2021},
  eprint = {2106.02435},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.02435},
}
Figure