Abstract
Neural vulnerability detection is a critical task in software security, yet existing Large Language Models (LLMs) often struggle with the structural nuances of code and high resource requirements. We present K-ASTRO, a lightweight, structure-aware Transformer model designed for efficient and generalizable vulnerability detection. By integrating semantic embeddings from LLMs with Abstract Syntax Tree (AST) structural logic, K-ASTRO achieves state-of-the-art performance on major benchmarks while remaining small enough to run on standard CPUs.
Introduction
Detecting security vulnerabilities in source code is often compared to finding a needle in a haystack. While Large Language Models (LLMs) have demonstrated impressive capabilities in code understanding, they face significant hurdles in security contexts. First, the resource intensity of massive models makes them impractical for local, real-time developer environments. Second, LLMs often exhibit "structural blindness," failing to track logical flows—such as variable sanitization—that are essential for identifying bugs. K-ASTRO addresses these challenges by combining the semantic power of LLMs with the rigid structural logic of compilers.
Methodology
K-ASTRO is built as a single-layer, encoder-only Transformer with approximately 1 million parameters. Its architecture is defined by three primary innovations: joint LLM adaptation, structure-aware attention, and diversity-introducing AST augmentation.
Joint LLM Adaptation and Structure-Aware Attention
The model leverages semantic embeddings from OpenAI’s text-embedding-3-small to capture high-level code intent. These are concatenated with structural features derived from the code's Abstract Syntax Tree (AST). To ensure the model respects the program's logic, K-ASTRO employs a structure-aware attention bias. This mechanism uses a co-occurrence matrix from the AST to force the model to prioritize relationships between structurally related nodes, such as a variable's declaration and its subsequent use in a sensitive function call.
Diversity-Introducing AST Augmentation
To improve generalizability, K-ASTRO utilizes a novel augmentation strategy inspired by mutation testing. A library of code subtrees is constructed from the training corpus. During training, the model randomly swaps components of a function's AST with subtrees from other functions sharing the same vulnerability label. This process teaches the model to identify the core characteristics of a vulnerability regardless of minor structural variations.
Experimental Setup
The model was evaluated using three rigorous C/C++ benchmarks:
- BigVul: Real-world CVE entries.
- DiverseVul: A large-scale dataset designed to minimize heuristic biases.
- PrimeVul: A high-quality dataset featuring chronological splitting and strict de-duplication to prevent data leakage.
Results and Key Insights
K-ASTRO demonstrates that inductive bias can outperform sheer scale. Despite having only 1 million parameters, it significantly outperformed GPT-4o on the BigVul CWE classification task, achieving a weighted F1 score of 76.31 compared to GPT-4o's 9.83.
Beyond accuracy, K-ASTRO is optimized for efficiency. It processes nearly 13,000 samples per second on a standard CPU and occupies only 4MB of disk space. This efficiency enables private, local code scanning without the need for expensive GPU clusters or cloud-based processing.
Conclusion
K-ASTRO represents a shift toward lightweight, specialized models for software engineering tasks. By baking structural knowledge into the architecture, the model achieves high generalizability and performance. Future work will focus on extending the model to inter-procedural analysis and expanding support to additional programming languages like Java and Python.
Summary updated