EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation

Tao Ge, Furu Wei

Abstract

We introduce EdgeFormer—a parameter-efficient Transformer for on-device seq2seq generation under strict computation and memory constraints. Compared with previous parameter-efficient Transformers, EdgeFormer applies two novel principles for cost-effective parameterization, allowing it to perform better given the same parameter budget. Moreover, EdgeFormer is further enhanced by layer adaptation innovation proposed for improving networks with shared layers. Extensive experiments show EdgeFormer can effectively outperform previous parameter-efficient Transformer baselines and achieve competitive results under both computation and memory constraints. Given these promising results, we release EdgeLM—the pretrained version of EdgeFormer—facilitating on-device seq2seq generation in practice.

1. Introduction

Standard Transformer models are computationally expensive and memory-intensive. A typical "base" model requires over 45 million parameters, translating to hundreds of megabytes of memory. This footprint makes them unsuitable for "edge" devices like smartphones, which often operate under strict RAM limits (e.g., <50MB) and latency requirements (<100ms).

While architectures like the Universal Transformer (UT) attempt to reduce size through weight sharing, they often rely on naive, uniform sharing that treats every layer identically. This "one-size-fits-all" approach leads to parameter underutilization and performance degradation. EdgeFormer addresses these limitations through targeted architectural innovations and load-balanced parameterization.

2. Methodology

EdgeFormer departs from standard Transformer design through three primary innovations designed to maximize the utility of every parameter.

2.1 Interleaved Decoder with Shared FFNs

EdgeFormer utilizes a highly asymmetrical architecture, employing a deep encoder (12 layers) and a very shallow decoder (2 layers). To further save memory, it implements "interleaved" lightweight Feed-Forward Networks (FFNs) where the internal dimension is significantly reduced ($d_{ffn} = d/4$).

2.2 Cost-Effective Parameterization

The authors propose two core principles for parameter distribution:

  1. Encoder-favored: Empirical evidence suggests the encoder is more critical for performance in seq2seq tasks. EdgeFormer parameterizes the encoder heavily while forcing the decoder to reuse the encoder's attention modules.
  2. Load-balanced: In typical shared models, parameters are often used unevenly (e.g., used 12 times in the encoder but only twice in the decoder). EdgeFormer balances this "load" to ensure parameters are utilized uniformly, preventing representation saturation.

2.3 Layer Adaptation (LA)

To prevent shared weights from becoming too generic, EdgeFormer introduces Layer Adaptation (LA)—tiny, layer-specific modifications that allow shared weights to specialize for their position in the network stack.

Layer Adaptation (LA) techniques to specialize shared weights: (a) baseline shared layers, (b) Bias-LA, (c) Adapter-LA, and (d) Prefix-LA.

As shown in the figure above, the authors explored three variants:

  • Bias-LA: Only the bias terms are unique to each layer.
  • Adapter-LA: Employs small Low-Rank Adaptation (LoRA) blocks for each layer.
  • Prefix-LA: Adds learnable "prefix tokens" to the input of each layer.

3. Experimental Results

EdgeFormer was evaluated across multiple benchmarks, including WMT14 (Translation), CoNLL-14 (GEC), and XSum (Summarization).

3.1 Performance and Scaling

EdgeFormer (8.6M parameters) achieved 28.0 BLEU on WMT14 En-De, significantly outperforming the Universal Transformer (27.0 BLEU). The effectiveness of the "encoder-favored" principle and its scaling behavior are illustrated below.

Performance and scaling of EdgeFormer principles. (a) Comparison of parameter sharing strategies showing the superiority of the encoder-favored approach. (b) Scaling behavior against the Universal Transformer (UT) baseline.

Figure (a) demonstrates that sharing the decoder while keeping a full encoder (orange triangle) yields better performance than sharing the encoder (green triangle) for a similar parameter count. Figure (b) shows that this approach scales more effectively than the standard Universal Transformer as the number of layers increases.

3.2 On-Device Efficiency

When quantized to int8, EdgeFormer achieves a latency of approximately 100ms for a 20-30 token sentence on a mid-range mobile CPU (Snapdragon 855). Its compressed disk footprint is roughly 28MB, with a peak execution memory of 60MB, making it highly viable for real-world on-device deployment.

4. Conclusion

EdgeFormer demonstrates that intelligent weight sharing and minor layer-specific adaptations can bridge the gap between massive server-side models and constrained edge devices. By prioritizing the encoder and balancing parameter load, EdgeFormer achieves state-of-the-art performance for its size class. The release of EdgeLM provides a robust foundation for future on-device seq2seq applications.

Summary updated

Abstract

We introduce EdgeFormer -- a parameter-efficient Transformer for on-device seq2seq generation under the strict computation and memory constraints. Compared with the previous parameter-efficient Transformers, EdgeFormer applies two novel principles for cost-effective parameterization, allowing it to perform better given the same parameter budget; moreover, EdgeFormer is further enhanced by layer adaptation innovation that is proposed for improving the network with shared layers. Extensive experiments show EdgeFormer can effectively outperform previous parameter-efficient Transformer baselines and achieve competitive results under both the computation and memory constraints. Given the promising results, we release EdgeLM -- the pretrained version of EdgeFormer, which is the first publicly available pretrained on-device seq2seq model that can be easily fine-tuned for seq2seq tasks with strong results, facilitating on-device seq2seq generation in practice.
BibTeX
@inproceedings{ge2022edgeformer,
  author = {Tao Ge and Furu Wei},
  title = {{EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=mo0eWPZ1zlO},
}
Figure