Abstract
The increasing complexity of software systems has led to a surge in cybersecurity vulnerabilities, necessitating efficient and scalable solutions for vulnerability assessment. However, the deployment of large pre-trained models in real-world scenarios is hindered by their substantial computational and storage demands. To address this challenge, we propose a novel resource-efficient framework that integrates knowledge distillation and particle swarm optimization to enable automated vulnerability assessment. Our framework employs a two-stage approach: First, particle swarm optimization is utilized to optimize the architecture of a compact student model, balancing computational efficiency and model capacity. Second, knowledge distillation is applied to transfer critical vulnerability assessment knowledge from a large teacher model to the optimized student model. This process significantly reduces the model size while maintaining high performance. Experimental results on an enhanced MegaVul dataset, comprising 12,071 CVSS (Common Vulnerability Scoring System) v3 annotated vulnerabilities, demonstrate the effectiveness of our approach. Our approach achieves a 99.4% reduction in model size while retaining 89.3% of the original model's accuracy. Furthermore, it outperforms state-of-the-art baselines by 1.7% in accuracy with 60% fewer parameters. The framework also reduces training time by 72.1% and architecture search time by 34.88% compared to traditional genetic algorithms.
Introduction
Software Vulnerability Assessment (SVA) is the critical process of identifying and ranking security flaws, typically using the Common Vulnerability Scoring System (CVSS). While Large Language Models (LLMs) like CodeBERT have set new benchmarks in this domain, their deployment is often restricted by high computational costs, significant latency, and large storage requirements. These "heavyweight" models are difficult to integrate into real-time monitoring systems or resource-constrained environments like embedded systems and IDEs.
To address these challenges, we present PSO-KDVA, a framework designed to compress massive code-analysis models while maintaining high performance. By integrating Particle Swarm Optimization (PSO) for architecture search and Knowledge Distillation (KD) for training, the framework achieves a 99.4% reduction in model size, making on-device vulnerability assessment a practical reality.
Methodology: PSO-KDVA Framework
The PSO-KDVA framework operates in two distinct stages to bridge the "capacity gap" between large teacher models and compact student models.
1. Optimal Architecture Search via PSO
Instead of manually designing a small model, the framework employs Particle Swarm Optimization (PSO). In this context, each "particle" represents a potential model architecture defined by parameters such as the number of layers, hidden units, and attention heads. A custom fitness function evaluates these architectures by balancing computational complexity (GFLOPs) against the capacity gap relative to the teacher model. This automated search ensures the resulting student model is structurally optimized for the specific task of vulnerability assessment.
2. Knowledge Distillation (KD)
Once the optimal architecture is identified, Knowledge Distillation is used to transfer expertise from a large "teacher" model (e.g., CodeBERT) to the "student" model.
As shown in Figure 1, the teacher model processes data to generate "soft labels"—probabilistic predictions that contain richer information than binary classifications. The student model is trained to mimic these probabilities, effectively inheriting the teacher's reasoning capabilities. This approach also allows the framework to leverage unlabeled data, as the teacher provides the ground truth for training.
Experimental Setup
The framework was evaluated using an enhanced MegaVul dataset. While the original MegaVul dataset contained approximately 18,000 C/C++ records based on CVSS v2, the authors updated it to the CVSS v3 standard, resulting in a cleaned dataset of 12,071 vulnerabilities. To test cross-language generalization, the framework was also applied to a MegaVulJava dataset, ensuring the method's effectiveness across different programming environments.
Results and Discussion
The PSO-KDVA framework demonstrates significant improvements in efficiency and performance across several key metrics:
- Massive Compression: Model size was reduced from 476 MB (CodeBERT) to 3 MB, a 99.4% reduction.
- High Fidelity: Despite the tiny size, the student model retained 89.3% of the teacher's accuracy.
- Lightning Speed: Inference latency dropped from 1635 ms to 58 ms (a 96.4% speedup), making it viable for real-time use.
- Search Efficiency: PSO proved to be 34.88% faster than traditional Genetic Algorithms (GA) for architecture search.
Despite these gains, the model showed some sensitivity to class imbalance, with a more pronounced drop in the Matthews Correlation Coefficient (MCC) compared to raw accuracy. This suggests that while the student model captures general patterns well, it may lose some nuance in identifying rare vulnerability types.
Conclusion
The PSO-KDVA framework provides a scalable solution for deploying advanced vulnerability assessment models in resource-constrained environments. By combining automated architecture search with knowledge transfer, it bridges the gap between model complexity and practical utility. Future work will explore hybrid compression techniques, such as combining distillation with pruning and quantization, and expanding validation to dynamic languages like Python and JavaScript.
Summary updated