PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment

Zequan Liu, Yi Zhao, Ming Tan, Wei Zhu, Aaron Xuxiang Tian

Introduction

In industrial Model-as-a-Service (MaaS) environments, a single large language model (LLM) backbone often serves multiple tenants, each requiring specialized fine-tuning. While Low-Rank Adaptation (LoRA) is the current industry standard, it introduces significant latency in multi-tenant scenarios because unique adapter weights cannot be merged into the base model during inference. This necessitates additional matrix multiplications at every token generation step. Conversely, simpler methods like $(IA)^3$ offer speed but lack the expressiveness required for complex reasoning tasks.

To bridge this gap, we present PARA (Prompt Aware Representation Adjustment), a parameter-efficient fine-tuning (PEFT) method that matches or exceeds LoRA's expressiveness while maintaining the inference efficiency of vector-scaling methods.

Methodology: The Vector Generator

The core of PARA is the Vector Generator (VG), a lightweight module integrated into each Transformer layer. Unlike static PEFT methods that use fixed learnable parameters, PARA dynamically generates adjustment vectors based on the input prompt.

Architecture of the PARA Vector Generator and its integration into Transformer layers.

Dynamic Prompt-Awareness

The VG employs a bottleneck architecture consisting of a pooling layer (focusing on the final prompt token), a down-projection, GeLU activation, and an up-projection. It produces three specific scaling vectors: $l_q$, $l_v$, and $l_u$. These vectors perform element-wise multiplication on the Query (Q) and Value (V) states in the attention module, and the Up-projection (U) states in the feed-forward network.

Inference Efficiency and KV-Caching

A key advantage of PARA is its synergy with KV-caching. Since the adjustment vectors are derived solely from the prompt, they are computed once during the pre-fill stage and cached. During the decoding phase, these vectors are reused without further computation, effectively adding zero overhead to per-token generation. This allows PARA to bypass the per-step latency bottlenecks inherent in LoRA, achieving generation speeds of up to 32.8 tokens per second (tps) compared to LoRA's 25.1 tps.

Experimental Results

PARA was evaluated using Llama-2 (7B and 13B) and Gemma 2B across diverse benchmarks, including SQuAD, BoolQ, COPA, MMLU, and specialized tasks like HSM10K (math reasoning) and Q2SQL (text-to-SQL).

Performance comparison of PARA against Full Fine-Tuning and baseline PEFT methods.

Performance and Efficiency

As shown in the results, PARA consistently outperforms standard PEFT baselines while utilizing a smaller parameter budget. For instance, on the HSM10K math dataset, PARA achieved an accuracy of 56.3%, surpassing both LoRA (55.1%) and AdaLoRA (55.6%) with only 8.9 million tunable parameters.

Scalability

The method's effectiveness is robust across model sizes. Evaluations on Llama-2 13B and Gemma 2B confirmed that PARA maintains its performance edge over LoRA and $(IA)^3$ as the backbone architecture scales, demonstrating its viability for state-of-the-art LLMs.

Conclusion and Future Work

PARA offers a compelling alternative to LoRA for high-throughput MaaS deployments by combining dynamic, prompt-aware expressiveness with static-like inference efficiency. Future research will focus on evaluating PARA at extreme scales (e.g., 70B+ parameters) and expanding its application to a broader range of NLP tasks such as summarization and information extraction.

Summary updated

BibTeX
@inproceedings{liu2024para,
  author = {Zequan Liu and Yi Zhao and Ming Tan and Wei Zhu and Aaron Xuxiang Tian},
  title = {{PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure