Introduction
Adapting large-scale Pre-trained Language Models (PLMs) like BERT or RoBERTa to downstream tasks typically involves full fine-tuning, where all model parameters are updated. While effective, this approach is computationally expensive and requires storing a full copy of the model for every task. To mitigate this, researchers introduced Prompt Tuning, which keeps the model frozen and learns a small "trigger" sequence. However, prompt tuning often suffers from a performance gap compared to full fine-tuning and exhibits training instability due to the "bottleneck" effect as prompts pass through deep Transformer layers.
The FL-Tuning Approach
This paper introduces FL-Tuning (Feed-Forward Layer Tuning), a method that moves beyond input-level prompts by injecting learnable parameters directly into the internal layers of the Transformer. Specifically, FL-Tuning targets the Feed-Forward Network (FFN) sub-layers, which account for approximately two-thirds of a Transformer's parameters and are often considered the model's "knowledge center."
Technical Implementation: The addFFN
The core innovation of FL-Tuning is the expansion of the hidden layer width. The authors mathematically demonstrate that adding new hidden units to a frozen weight matrix is functionally equivalent to adding a small, parallel FFN—termed addFFN—alongside the original frozen FFN.
During training, the original Transformer parameters remain entirely frozen. Only the parameters within the addFFN are updated. At inference time, the output is the simple summation of the frozen original FFN and the task-specific addFFN. This "side-car" architecture allows for efficient adaptation without resizing massive frozen tensors.
Experimental Setup and Datasets
FL-Tuning was evaluated using the CLUE (Chinese Language Understanding Evaluation) benchmark. The evaluation spanned 11 diverse datasets, including text classification (IFLYTEK, TNEWS), semantic similarity (AFQMC), natural language inference (CMNLI, OCNLI), and machine reading comprehension (C3, CMRC2018).
Key Insights and Results
Efficiency and Performance
FL-Tuning achieves competitive results by training only approximately 3% of the total model parameters. It consistently outperforms state-of-the-art prompt tuning methods like P-tuning v2. Notably, on the WSC 1.0 task, FL-Tuning improved accuracy by 17.93% over prompt-based alternatives. In several instances, FL-Tuning even surpassed the performance of full fine-tuning, such as a 12.9% accuracy gain on the WSC 1.1 dataset.
Robustness and Convergence
Unlike prompt tuning, which is highly sensitive to learning rate and initialization, FL-Tuning demonstrates significant hyperparameter robustness. It achieves high performance across a wider range of settings and converges approximately 1.17 times faster than P-tuning v2, reducing overall compute costs.
Layer Sensitivity
The researchers observed that the position of the tuned layers matters; adding hidden units to the deeper layers of the Transformer provides a more substantial performance boost than modifications to the earlier layers.
Conclusion
FL-Tuning represents a shift from input-level adaptation to internal layer adaptation. By targeting the FFN layers with a mathematically elegant parallel structure, it offers a more stable, efficient, and high-performing alternative to both prompt tuning and full fine-tuning. Future work may explore hybrid approaches that simultaneously tune both Multi-head Attention and FFN layers to further close the performance gap in complex tasks like Machine Reading Comprehension.
Summary updated