Learn-to-Share: A Hardware-friendly Transfer Learning Framework Exploiting Computation and Parameter Sharing

Cheng Fu, Hanxian Huang, Xinyun Chen, Yuandong Tian, Jishen ZhaoICML 2021 · OpenReview

TL;DR

LeTS (Learn-to-Share) is a novel transfer learning framework that addresses the massive computational and storage costs of fine-tuning Transformers for multiple tasks. By combining a fixed pre-trained backbone with a task-specific "computation sharing" scheme discovered via Neural Architecture Search (NAS), LeTS reduces total operations by nearly 50% with only a 0.2% drop in accuracy.

What Problem Does This Paper Address?

Fine-tuning pre-trained models like BERT has become the industry standard for achieving high performance in Natural Language Processing (NLP). However, this approach has a major scalability flaw: as the number of tasks grows, the requirements for both storage and computation grow linearly.

While previous research (like "Adapters" or "DiffPruning") successfully reduced the storage footprint by sharing parameters, they didn't solve the computation bottleneck. Even if you share 99% of the parameters, you still have to perform a full forward pass through the entire model for every single task. This is a dealbreaker for real-time applications on mobile devices or high-throughput cloud services where latency and power consumption are critical.

Methods

The authors introduce LeTS, which treats fine-tuning as a search for the most efficient way to reuse existing "cached" computations from a fixed pre-trained model.

LeTS computation sharing architecture
  1. Fixed Backbone + Additive Components: LeTS keeps the original Transformer weights frozen. It adds small, learnable components that "plug into" the intermediate activations of the fixed model.
  2. Computation Sharing via DNAS: The framework uses Differentiable Neural Architecture Search (DNAS) to decide, for each task, whether to use the output of a fine-tuned layer or simply reuse the "cached" output from the frozen pre-trained layer. If a task reuses a cached result, it completely bypasses the computation for that specific fine-tuned layer.
  3. Delta-Pruning: To keep the parameter count low, the authors propose "Delta-Pruning." Instead of pruning the entire model, they only prune the difference ($\delta$) between the pre-trained weights and the fine-tuned weights. This happens at an early stage, ensuring the task-specific overhead remains tiny (around 1.4%).
  4. Decoupled Execution: By using a Bi-LSTM and a pooling layer to aggregate outputs from various Transformer layers, LeTS breaks the strict sequential dependency of standard models. This allows for concurrent execution (parallelism) within the Transformer, further speeding up inference.

Datasets

The framework was evaluated on the GLUE (General Language Understanding Evaluation) benchmark, which includes nine diverse NLP tasks such as:

  • CoLA (Linguistic acceptability)
  • SST-2 (Sentiment analysis)
  • MRPC & QQP (Paraphrase detection)
  • STS-B (Semantic similarity)
  • MNLI, QNLI, & RTE (Natural language inference)

The authors compared LeTS against standard BERT-Large fine-tuning, parameter-efficient methods like Adapters and DiffPruning, and compressed models like DistilBERT and TinyBERT.

Key Insights and Ideas

  • Computation is the New Storage: The authors argue that in modern hardware environments, reducing FLOPs (floating-point operations) is often more important than reducing parameter counts, as computation directly impacts battery life and latency.
  • Not All Layers Are Equal: The NAS results show that different tasks rely on different layers. Some tasks can "skip" fine-tuning on bottom layers and still achieve peak accuracy by only tuning the top, while others benefit from sparse updates throughout.
  • Exploiting "Free" Activations: Because the pre-trained model is fixed and shared across all tasks, its activations only need to be calculated once. LeTS treats these as a "free" resource that sub-tasks can tap into whenever they don't need a specialized, fine-tuned feature.

Key Results

GLUE Score vs GFLOPs comparison
  • Efficiency: On the GLUE benchmark, LeTS reduced computation by 49.5% compared to full fine-tuning.
  • Accuracy: This massive reduction came at a negligible cost—only a 0.2% average accuracy loss.
  • Parameter Overhead: LeTS adds only 1.4% extra parameters per task, making it highly competitive with state-of-the-art parameter-sharing methods like Adapters, but with much better latency.
  • Speedup: In specific configurations (like LeTS-G-4), the model achieved a 3.84x max speedup by enabling parallel execution of layer groups.

Limitations and Future Work

The current version of LeTS is designed for "streaming" tasks—where tasks arrive one by one. While it excels at sharing computation from a fixed backbone, it doesn't yet fully explore cross-task sharing (where Task A might reuse a fine-tuned layer specifically created for Task B).

The authors also note that while LeTS is "orthogonal" to model compression (meaning you could theoretically use LeTS on top of a DistilBERT backbone), they haven't yet fully integrated these two approaches into a single unified pipeline. Future work could involve combining LeTS with multi-task learning (MTL) to see if joint training can further improve the "sharing policies" discovered by the NAS.

Summary updated

BibTeX
@inproceedings{fu2021learn,
  author = {Cheng Fu and Hanxian Huang and Xinyun Chen and Yuandong Tian and Jishen Zhao},
  title = {{Learn-to-Share: A Hardware-friendly Transfer Learning Framework Exploiting Computation and Parameter Sharing}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=-9qDr-TS0l},
}
Figure