Introduction
In industrial Model-as-a-Service (MaaS) environments, a single large language model (LLM) backbone often serves multiple tenants, each requiring specialized fine-tuning. While Low-Rank Adaptation (LoRA) is the current industry standard, it introduces significant latency in multi-tenant scenarios because unique adapter weights cannot be merged into the base model during inference. This necessitates additional matrix multiplications at every token generation step. Conversely, simpler methods like $(IA)^3$ offer speed but lack the expressiveness required for complex reasoning tasks.
To bridge this gap, we present PARA (Prompt Aware Representation Adjustment), a parameter-efficient fine-tuning (PEFT) method that matches or exceeds LoRA's expressiveness while maintaining the inference efficiency of vector-scaling methods.
Methodology: The Vector Generator
The core of PARA is the Vector Generator (VG), a lightweight module integrated into each Transformer layer. Unlike static PEFT methods that use fixed learnable parameters, PARA dynamically generates adjustment vectors based on the input prompt.
Dynamic Prompt-Awareness
The VG employs a bottleneck architecture consisting of a pooling layer (focusing on the final prompt token), a down-projection, GeLU activation, and an up-projection. It produces three specific scaling vectors: $l_q$, $l_v$, and $l_u$. These vectors perform element-wise multiplication on the Query (Q) and Value (V) states in the attention module, and the Up-projection (U) states in the feed-forward network.
Inference Efficiency and KV-Caching
A key advantage of PARA is its synergy with KV-caching. Since the adjustment vectors are derived solely from the prompt, they are computed once during the pre-fill stage and cached. During the decoding phase, these vectors are reused without further computation, effectively adding zero overhead to per-token generation. This allows PARA to bypass the per-step latency bottlenecks inherent in LoRA, achieving generation speeds of up to 32.8 tokens per second (tps) compared to LoRA's 25.1 tps.
Experimental Results
PARA was evaluated using Llama-2 (7B and 13B) and Gemma 2B across diverse benchmarks, including SQuAD, BoolQ, COPA, MMLU, and specialized tasks like HSM10K (math reasoning) and Q2SQL (text-to-SQL).
Performance and Efficiency
As shown in the results, PARA consistently outperforms standard PEFT baselines while utilizing a smaller parameter budget. For instance, on the HSM10K math dataset, PARA achieved an accuracy of 56.3%, surpassing both LoRA (55.1%) and AdaLoRA (55.6%) with only 8.9 million tunable parameters.
Scalability
The method's effectiveness is robust across model sizes. Evaluations on Llama-2 13B and Gemma 2B confirmed that PARA maintains its performance edge over LoRA and $(IA)^3$ as the backbone architecture scales, demonstrating its viability for state-of-the-art LLMs.
Conclusion and Future Work
PARA offers a compelling alternative to LoRA for high-throughput MaaS deployments by combining dynamic, prompt-aware expressiveness with static-like inference efficiency. Future research will focus on evaluating PARA at extreme scales (e.g., 70B+ parameters) and expanding its application to a broader range of NLP tasks such as summarization and information extraction.
Summary updated