EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation

Tao Ge, Si-Qing Chen, Furu Wei

Abstract

EdgeFormer is a parameter-efficient Transformer designed specifically for high-performance sequence-to-sequence (seq2seq) generation on resource-constrained edge devices. By utilizing novel "encoder-favored" and "load-balanced" parameterization strategies alongside lightweight layer adaptation, it achieves state-of-the-art results for models under 10 million parameters, maintaining sub-100ms latency and a memory footprint under 50MB.

Introduction

While Large Language Models (LLMs) dominate the headlines, deploying them on-device (phones, IoT, etc.) remains a massive challenge. Standard Transformer architectures are "parameter-hungry" and memory-intensive. Previous "efficient" Transformers often focused solely on reducing FLOPs (computational cost) but failed to address the strict memory and latency requirements of real-world edge hardware.

Specifically, in seq2seq tasks like translation or summarization, the decoder often becomes a bottleneck. Existing parameter-sharing methods, such as the Universal Transformer, often distribute parameters inefficiently, leading to a "one-size-fits-all" model that lacks the specialized nuance required for high-quality generation.

Methods

EdgeFormer introduces a structural shift and two core parameterization principles to optimize for edge deployment:

1. Interleaved Decoder with Shared Lightweight FFNs

Unlike the vanilla Transformer where the Feed-Forward Network (FFN) is much wider than the model dimension ($d_{ffn} > d$), EdgeFormer uses a shallow decoder with interleaved attention modules and shared, lightweight FFNs ($d_{ffn} < d$). This structural change significantly reduces the memory footprint without sacrificing the model's ability to process complex sequences.

Comparison between (a) vanilla Transformer and (b) EdgeFormer interleaved architecture.

2. Encoder-favored and Load-balanced Parameterization

The authors argue that the encoder is more critical for performance. They allocate the bulk of the parameter budget to the encoder and force the decoder to "reuse" the encoder’s attention parameters. Furthermore, they introduce Load-balanced Parameterization, ensuring each parameter is "exploited" an equal number of times during a forward pass. This prevents the dilution of feature learning that occurs when specific weights are over- or under-used.

3. Layer Adaptation (LA)

To compensate for the loss of specialization caused by weight sharing, EdgeFormer applies lightweight adaptation techniques. Three versions were tested:

  • Bias-LA: Untying only the bias terms.
  • Adapter-LA: Using a LoRA-style low-rank adaptation block.
  • Prefix-LA: Using learnable "prefix" tokens (prefix-tuning).

Key Results

EdgeFormer was evaluated on standard NLP benchmarks including WMT14 (Machine Translation), CoNLL-14 (GEC), and XSum (Summarization).

  • Efficiency and Latency: EdgeFormer operates with fewer than 10 million parameters and fits within a 50MB RAM footprint. On a mid-to-high-end mobile CPU (Snapdragon 855), it achieves a latency of ~100ms for average sentence lengths.
  • Performance: On WMT14 En-De, EdgeFormer (8.6M params) achieved 26.9 BLEU, outperforming the Universal Transformer (7.4M params, 26.2 BLEU) and rivaling much larger models.
  • Pretraining Gains: The pretrained version, EdgeLM, showed massive improvements across downstream tasks, particularly in summarization and question generation, proving that pretraining is highly effective even for tiny, on-device models.

Discussion and Future Work

The success of EdgeFormer highlights that the decoder is often over-parameterized in standard models and that balancing the parameter "load" across the network leads to significant performance gains.

However, the authors acknowledge that their current load-balancing mechanism is somewhat naive, assuming all modules require the same capacity. Future work involves developing more sophisticated, dynamic load-balancing strategies and researching how different layers might benefit from varying amounts of adaptation.

Summary updated

Abstract

We introduce EdgeFormer -- a parameter-efficient Transformer for on-device seq2seq generation under the strict computation and memory constraints. Compared with the previous parameter-efficient Transformers, EdgeFormer applies two novel principles for cost-effective parameterization, allowing it to perform better given the same parameter budget; moreover, EdgeFormer is further enhanced by layer adaptation innovation that is proposed for improving the network with shared layers. Extensive experiments show EdgeFormer can effectively outperform previous parameter-efficient Transformer baselines and achieve competitive results under both the computation and memory constraints. Given the promising results, we release EdgeLM -- the pretrained version of EdgeFormer, which is the first publicly available pretrained on-device seq2seq model that can be easily fine-tuned for seq2seq tasks with strong results, facilitating on-device seq2seq generation in practice.
BibTeX
@inproceedings{ge2022edgeformer,
  author = {Tao Ge and Si-Qing Chen and Furu Wei},
  title = {{EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=mo0eWPZ1zlO},
}
Figure