FL-Tuning: Layer Tuning for Feed-Forward Network in Transformer

Jingping Liu, Yuqiu Song, Kui Xue, Hongli Sun, Chao Wang, Lihan Chen, Haiyun Jiang, Jiaqing Liang, Tong RuanarXiv 2022 · DOI · arXiv

Introduction

Adapting large-scale Pre-trained Language Models (PLMs) like BERT or RoBERTa to downstream tasks typically involves full fine-tuning, where all model parameters are updated. While effective, this approach is computationally expensive and requires storing a full copy of the model for every task. To mitigate this, researchers introduced Prompt Tuning, which keeps the model frozen and learns a small "trigger" sequence. However, prompt tuning often suffers from a performance gap compared to full fine-tuning and exhibits training instability due to the "bottleneck" effect as prompts pass through deep Transformer layers.

The FL-Tuning Approach

This paper introduces FL-Tuning (Feed-Forward Layer Tuning), a method that moves beyond input-level prompts by injecting learnable parameters directly into the internal layers of the Transformer. Specifically, FL-Tuning targets the Feed-Forward Network (FFN) sub-layers, which account for approximately two-thirds of a Transformer's parameters and are often considered the model's "knowledge center."

Comparison of FL-tuning with traditional discrete and continuous prompt tuning methods.

Technical Implementation: The addFFN

The core innovation of FL-Tuning is the expansion of the hidden layer width. The authors mathematically demonstrate that adding new hidden units to a frozen weight matrix is functionally equivalent to adding a small, parallel FFN—termed addFFN—alongside the original frozen FFN.

During training, the original Transformer parameters remain entirely frozen. Only the parameters within the addFFN are updated. At inference time, the output is the simple summation of the frozen original FFN and the task-specific addFFN. This "side-car" architecture allows for efficient adaptation without resizing massive frozen tensors.

Detailed architecture of FL-tuning showing the parallel addFFN structure within a Transformer layer.

Experimental Setup and Datasets

FL-Tuning was evaluated using the CLUE (Chinese Language Understanding Evaluation) benchmark. The evaluation spanned 11 diverse datasets, including text classification (IFLYTEK, TNEWS), semantic similarity (AFQMC), natural language inference (CMNLI, OCNLI), and machine reading comprehension (C3, CMRC2018).

Key Insights and Results

Efficiency and Performance

FL-Tuning achieves competitive results by training only approximately 3% of the total model parameters. It consistently outperforms state-of-the-art prompt tuning methods like P-tuning v2. Notably, on the WSC 1.0 task, FL-Tuning improved accuracy by 17.93% over prompt-based alternatives. In several instances, FL-Tuning even surpassed the performance of full fine-tuning, such as a 12.9% accuracy gain on the WSC 1.1 dataset.

Robustness and Convergence

Unlike prompt tuning, which is highly sensitive to learning rate and initialization, FL-Tuning demonstrates significant hyperparameter robustness. It achieves high performance across a wider range of settings and converges approximately 1.17 times faster than P-tuning v2, reducing overall compute costs.

Layer Sensitivity

The researchers observed that the position of the tuned layers matters; adding hidden units to the deeper layers of the Transformer provides a more substantial performance boost than modifications to the earlier layers.

Conclusion

FL-Tuning represents a shift from input-level adaptation to internal layer adaptation. By targeting the FFN layers with a mathematically elegant parallel structure, it offers a more stable, efficient, and high-performing alternative to both prompt tuning and full fine-tuning. Future work may explore hybrid approaches that simultaneously tune both Multi-head Attention and FFN layers to further close the performance gap in complex tasks like Machine Reading Comprehension.

Summary updated

Abstract

Prompt tuning is an emerging way of adapting pre-trained language models to downstream tasks. However, the existing studies are mainly to add prompts to the input sequence. This way would not work as expected due to the intermediate multi-head self-attention and feed-forward network computation, making model optimization not very smooth. Hence, we propose a novel tuning way called layer tuning, aiming to add learnable parameters in Transformer layers. Specifically, we focus on layer tuning for feed-forward network in the Transformer, namely FL-tuning. It introduces additional units into the hidden layer of each feed-forward network. We conduct extensive experiments on the public CLUE benchmark. The results show that: 1) Our FL-tuning outperforms prompt tuning methods under both full-data and few-shot settings in almost all cases. In particular, it improves accuracy by 17.93% (full-data setting) on WSC 1.0 and F1 by 16.142% (few-shot setting) on CLUENER over P-tuning v2. 2) Our FL-tuning is more stable and converges about 1.17 times faster than P-tuning v2. 3) With only about 3% of Transformer's parameters to be trained, FL-tuning is comparable with fine-tuning on most datasets, and significantly outperforms fine-tuning (e.g., accuracy improved by 12.9% on WSC 1.1) on several datasets. The source codes are available at https://github.com/genggui001/FL-Tuning.
BibTeX
@misc{liu2022fl,
  author = {Jingping Liu and Yuqiu Song and Kui Xue and Hongli Sun and Chao Wang and Lihan Chen and Haiyun Jiang and Jiaqing Liang and Tong Ruan},
  title = {{FL-Tuning: Layer Tuning for Feed-Forward Network in Transformer}},
  year = {2022},
  eprint = {2206.15312},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.15312},
}
Figure