An Adaptive Language-Agnostic Pruning Method for Greener Language Models for Code

TL;DR

This paper introduces ALPINE, a novel method to make large language models (LLMs) for code more efficient by adaptively pruning less important tokens from input sequences. ALPINE significantly reduces computational costs (FLOPs), memory usage, and CO2 emissions, all while maintaining high predictive performance, making LLMs for code more accessible and sustainable.

What Problem Does This Paper Address?

Large language models (LLMs) have revolutionized software engineering tasks, but their immense computational requirements for training and inference pose significant challenges. These demands translate to high energy consumption, a substantial carbon footprint, and increased financial investment, limiting their accessibility for researchers and developers with limited resources. Existing methods for reducing model size or computational load often compromise performance or lack generality. This paper tackles the problem of making these powerful code LLMs more "green" and efficient without sacrificing their effectiveness.

Methods

The core of this work is ALPINE, an adaptive, language-agnostic pruning technique designed to be seamlessly integrated into any Transformer-based model. ALPINE operates by inserting a pruning module within each Transformer layer, specifically between the Multi-Head Attention (MHA) and Feed-Forward Network (FFNN) blocks.

Here's how it works:

1. Importance Scoring: ALPINE calculates an "importance score" for each token in the input sequence. This score is derived from the attention probabilities output by the MHA layer, essentially quantifying how much attention a token receives.
2. Adaptive Pruning: Based on these scores, tokens are pruned if they fall outside a dynamically determined range (defined by the mean and standard deviation of scores, controlled by a hyperparameter $\alpha$). This range is adjusted per layer and per model.
3. Sequence Compression: Pruned tokens are either entirely removed or merged into a single representation. This process effectively reduces the input sequence length, leading to fewer computations in subsequent layers, particularly the computationally intensive FFNN.
4. Language-Agnostic: A key feature is its independence from specific programming language syntax or semantics, allowing it to work with any code represented as a sequence of tokens.

Datasets

The study evaluates ALPINE on two software engineering tasks using established datasets:

• Defect Prediction: The DEVIGN dataset, curated from four large-scale open-source C repositories, containing manually annotated code snippets for various defects.
• Code Clone Detection: A filtered version of the BigCloneBenchmark dataset, focusing on Java code snippets, used for binary classification to predict whether two snippets are clones.

Key Insights and Ideas

• "Less is More" for Code LLMs: Just as compact models show not all parameters are needed, ALPINE demonstrates that not all tokens in a code sequence are critical for accurate predictions. This principle of task-specific efficiency is crucial for sustainable AI.
• FFNN Bottleneck: The analysis confirms that the Feed-Forward Network (FFNN) layer is a significant computational bottleneck in Transformer models for code. Pruning before this layer effectively reduces its workload.
• Layer-Specific Pruning Matters: Pruning at even-indexed layers often yielded better results than pruning at odd-indexed layers, suggesting that early compression can have a cascading positive effect.
• Merging vs. Dropping: Merging pruned tokens into a single representation consistently led to better performance than completely discarding them, indicating that some information from less important tokens can still be beneficial.
• Consumer GPU Potential: ALPINE shows a more substantial efficiency improvement on consumer-grade GPUs (RTX2080) compared to high-performance GPUs (A100), highlighting its potential to democratize the use of powerful code LLMs.

Key Results

• Computational Efficiency: ALPINE achieved an average reduction of ×2 in FLOPs and up to 58.1% decrease in memory footprint across models and tasks.
• Performance Retention: Despite significant compression, ALPINE maintained an average of 98.15% of the original predictive performance, with some instances even showing slight improvements.
• Throughput Improvement: Model throughput increased by up to 51.6%, enabling faster processing.
• CO2 Emission Reduction: ALPINE led to a substantial reduction in CO2 emissions, with a notable 44.85% reduction on the RTX2080 GPU.
• Sequence Compression: Input sequences were compressed to approximately ×3 less than their original length in some scenarios.

Limitations and Future Work

• Model Architectures: The current study focuses on encoder-only Transformer models. The authors plan to extend ALPINE to encoder-decoder and decoder-only architectures in future work.
• Scoring Functions: Investigating alternative importance scoring functions beyond attention-based methods is a direction for future research.
• Merging Strategies: Exploring more adaptable merging strategies for pruned tokens could further enhance performance.
• Task Dependency: The effectiveness of pruning strategies (e.g., even vs. odd layers) can be task-dependent, suggesting room for more fine-grained adaptive approaches.

@article{saad2025adaptive,
  author = {Mootez Saad and José Antonio Hernández López and Boqi Chen and Dániel Varró and Tushar Sharma},
  title = {{An Adaptive Language-Agnostic Pruning Method for Greener Language Models for Code}},
  journal = {Proc. ACM Softw. eng},
  year = {2025},
  doi = {10.1145/3715773},
}