Introduction
Large Language Models (LLMs) have demonstrated significant potential in identifying security vulnerabilities within source code. However, deploying these models for static analysis involves a substantial "computational tax." Fine-tuning massive models requires extensive GPU resources, energy, and memory. While compression techniques such as pruning and quantization can mitigate these costs, they often introduce unpredictable performance degradation. Traditionally, the efficacy of a compression strategy is only known after the resource-intensive fine-tuning process is complete.
This paper introduces LPASS (Linear Probes As Stepping Stones), a framework designed to predict the performance of compressed LLMs early in the pipeline. By utilizing lightweight linear probes to analyze internal model activations, LPASS enables informed decisions regarding model pruning and quantization before full-scale fine-tuning.
The LPASS Framework
The core innovation of LPASS is the use of Linear Probes (LPs)—simple, single-layer neural networks—to monitor how information is processed across the layers of a pre-trained LLM.
The methodology follows a three-step optimization process:
- Feature Extraction: The framework identifies code metrics, specifically Cyclomatic Complexity (CC) and Halstead Difficulty (HD), which serve as proxies for vulnerability-related information.
- Probing: LPs are trained to predict these features using the internal activations at every layer of the LLM. This allows researchers to visualize the "knowledge acquisition" curve of the model.
- Optimization:
- Layer Pruning: A cut-off layer ($k_{cut}$) is identified where the model's ability to predict code features plateaus. Layers beyond this point are considered redundant and discarded.
- Performance Estimation: The accuracy of these probes is used to estimate the final F1-score of the fine-tuned, compressed model, allowing for rapid iteration.
Experimental Evaluation
The researchers evaluated LPASS using two distinct architectures: BERT (encoder-only) and Gemma (decoder-only). The evaluation leveraged three large-scale C/C++ vulnerability datasets—DiverseVul, Big-Vul, and PrimeVul—comprising approximately 480,000 samples across the 12 most critical CWE categories.
Key Findings
The study yielded several significant insights into the behavior of LLMs in security contexts:
- Significant Redundancy: For the Gemma model, LPASS revealed that 72.2% of layers could be removed without compromising precision. This suggests that deep, general-purpose layers may not contribute meaningfully to structural code analysis tasks.
- Efficiency and Speed: The linear probes can be computed in under 150 seconds, providing a performance estimate with an average error of only 8.68%. This "early warning system" drastically reduces the search space for optimal compression.
- Compression as Regularization: In some instances, compressed models outperformed their full-sized counterparts. For example, a pruned Gemma model achieved a 1.6% improvement in F1-score, suggesting that removing redundant layers can reduce noise and improve generalization.
- Resource Savings: For Gemma, LPASS enabled a 42.98% reduction in model size, a 29.4% reduction in training time, and a 23.8% reduction in inference time, while maintaining an 86.9% accuracy in multi-class detection.
Conclusion and Future Work
LPASS demonstrates that linear probes are effective "stepping stones" for navigating the trade-offs between LLM performance and efficiency. By identifying redundant layers early, the framework makes advanced vulnerability detection more accessible for resource-constrained environments.
Future research will explore the applicability of LPASS to other programming languages like Python and Java, investigate non-linear relationships for even more precise performance estimation, and test the framework on non-security tasks such as code summarization.
Summary updated