GRASP: GRouped Activation Shared Parameterization for Parameter-Efficient Fine-Tuning and Robust Inference of Transformers

Malyaban Bal, Abhronil SenguptaarXiv 2025 · DOI · arXiv

Abstract

Fine-tuning Large Language Models (LLMs) for domain-specific tasks remains a significant computational challenge. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA reduce the number of trainable parameters, they often still scale with the model's hidden dimension. Furthermore, deploying these models on edge AI hardware introduces susceptibility to physical noise. This paper presents GRASP (GRouped Activation Shared Parameterization), an ultra-lightweight PEFT method that achieves extreme parameter compression by sharing scaling and shifting parameters across grouped activation dimensions. We further introduce StochGRASP, a stochastic variant that learns perturbation distributions to ensure robust inference on noisy hardware. Our results demonstrate that GRASP can achieve competitive performance with up to 75× fewer parameters than LoRA, while StochGRASP maintains high accuracy under simulated hardware noise where deterministic methods fail.

Introduction

The rapid advancement of Large Language Models (LLMs) has revolutionized natural language processing, yet the sheer size of these models makes full fine-tuning prohibitively expensive for most applications. Parameter-Efficient Fine-Tuning (PEFT) has emerged as a solution, focusing updates on a small subset of parameters. However, as models grow, even "efficient" methods like LoRA or Adapters can require millions of parameters, scaling linearly with the hidden dimension $D$. Additionally, the transition from cloud-based inference to edge deployment on specialized AI hardware (e.g., analog non-volatile memory) introduces a new hurdle: hardware-level noise.

Addressing the Efficiency-Robustness Gap

Standard PEFT methods are typically deterministic and do not account for the variability inherent in physical hardware. This paper addresses two critical needs: (1) reducing the parameter footprint of PEFT beyond current state-of-the-art methods, and (2) ensuring that fine-tuned models remain functional when deployed on noisy edge devices.

Methodology

The authors propose a two-stage approach to address these challenges, moving from extreme compression to hardware-aware robustness.

1. GRASP (GRouped Activation Shared Parameterization)

GRASP optimizes the activation space rather than the weight space directly. Instead of learning unique scaling ($\gamma$) and shifting ($\beta$) parameters for every dimension of a layer's hidden state, GRASP partitions the $D$ dimensions into $K$ groups ($K \ll D$). All dimensions within a group share the same $\gamma$ and $\beta$ values. This reduces the trainable parameter count per layer from $O(D)$ to $O(K)$, introducing no additional matrix multiplications during training.

Overview of the GRASP and StochGRASP frameworks for parameter-efficient training and robust inference.

2. StochGRASP (Stochastic GRASP)

Building on the efficiency of GRASP, StochGRASP introduces a probabilistic framework. Instead of learning fixed scalar values for updates, it learns the parameters of a Gaussian distribution (mean $\mu$ and standard deviation $\sigma$) for weight perturbations. During training, a noise-aware loss function is employed to regularize the learned $\sigma$ toward a target noise profile. This "internalizes" the expected hardware variability, making the model inherently resilient to inference-time noise.

Key Insights: Emergent Multimodality

A fascinating observation from the study of GRASP is the behavior of learned parameters under extreme compression. As the number of groups $K$ decreases, the model compensates for the lack of capacity by developing multimodal parameter distributions.

The grouping mechanism of GRASP (a) and the resulting multimodal distributions of shifting (b) and scaling (c) parameters under compression.

This multimodality suggests that the model discovers critical latent structures within the data, effectively "clustering" important features to maintain performance despite the restricted parameter budget. Mathematically, the authors demonstrate that group-wise activation shifts are functionally equivalent to introducing an effective bias in the frozen weight space, providing a theoretical bridge between activation modulation and weight-based PEFT.

Key Results

The methods were tested across Natural Language Understanding (GLUE benchmark) and Natural Language Generation (E2E NLG challenge) tasks.

  • Extreme Parameter Efficiency: On RoBERTa-base, GRASP with $K=32$ achieved an average GLUE score of 84.6% while using only 0.004% of the total parameters. This represents a 75× reduction in trainable parameters compared to LoRA and a 22.5× reduction compared to BitFit.
  • Generative Performance: In NLG tasks using GPT-2 Medium, GRASP ($K=64$) outperformed rank-1 LoRA while using 10× fewer parameters (0.003% vs 0.03%).
  • Hardware Noise Robustness: The impact of StochGRASP is most evident under noise. When subjected to simulated hardware noise ($\sigma = 0.01$), the accuracy of a deterministic GRASP model on SST-2 dropped to 61.5%. StochGRASP, however, maintained an accuracy of ≥ 86%, demonstrating its effectiveness for edge deployment.

Conclusion

GRASP and StochGRASP represent a significant step forward in making LLMs both more efficient to fine-tune and more robust for real-world deployment. By leveraging grouped activation sharing and stochastic parameterization, the authors provide a pathway for deploying high-performance language models on resource-constrained and noisy edge hardware. Future work will focus on integrating more complex, hardware-specific noise models to further refine these techniques for physical analog AI chips.

Summary updated

Abstract

Parameter-efficient fine-tuning (PEFT) provides a scalable alternative to full-model adaptation by updating only a small subset of parameters in large pre-trained models. We introduce GRASP - GRouped Activation Shared Parameterization - a lightweight PEFT framework that partitions the D-dimensional token representations of selected layers into K << D groups and learns a shared scaling and shifting vector for each group. This grouped modulation reduces the number of trainable parameters significantly while preserving the ability of the model to learn task-specific features. Building on this formulation, we further propose StochGRASP, which learns Gaussian distributions as perturbations to the pre-trained weights rather than deterministic values. This probabilistic parameterization along with a noise-aware loss function formulation enables modelling hardware-level variability in programmed weights and significantly improves robustness under non-ideal inference conditions-an important requirement for deployment on edge-based emerging AI hardware. Across GLUE (RoBERTa-base & RoBERTa-large) and E2E NLG (GPT-2 Medium), GRASP matches or exceeds the performance of established PEFT methods while achieving an order of magnitude reduction in trainable parameters compared to LoRA and BitFit. Under varying levels of noise, StochGRASP consistently outperforms deterministic variants, demonstrating its suitability for energy-efficient and noise-prone hardware platforms.
BibTeX
@misc{bal2025grasp,
  author = {Malyaban Bal and Abhronil Sengupta},
  title = {{GRASP: GRouped Activation Shared Parameterization for Parameter-Efficient Fine-Tuning and Robust Inference of Transformers}},
  year = {2025},
  eprint = {2512.04296},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.04296},
}
Figure