ASTRO: An AST-Assisted Approach for Generalizable Neural Clone Detection

Yifan Zhang, Junwen Yang, Haoyu Dong, Qingchen Wang, Huajie Shao, Kevin Leach, Yu Huang

Abstract

Neural vulnerability detection is a critical task in software security, yet existing Large Language Models (LLMs) often struggle with the structural nuances of code and high resource requirements. We present K-ASTRO, a lightweight, structure-aware Transformer model designed for efficient and generalizable vulnerability detection. By integrating semantic embeddings from LLMs with Abstract Syntax Tree (AST) structural logic, K-ASTRO achieves state-of-the-art performance on major benchmarks while remaining small enough to run on standard CPUs.

Introduction

Detecting security vulnerabilities in source code is often compared to finding a needle in a haystack. While Large Language Models (LLMs) have demonstrated impressive capabilities in code understanding, they face significant hurdles in security contexts. First, the resource intensity of massive models makes them impractical for local, real-time developer environments. Second, LLMs often exhibit "structural blindness," failing to track logical flows—such as variable sanitization—that are essential for identifying bugs. K-ASTRO addresses these challenges by combining the semantic power of LLMs with the rigid structural logic of compilers.

Methodology

K-ASTRO is built as a single-layer, encoder-only Transformer with approximately 1 million parameters. Its architecture is defined by three primary innovations: joint LLM adaptation, structure-aware attention, and diversity-introducing AST augmentation.

The K-ASTRO architecture combines LLM semantic embeddings with AST-based structural features.

Joint LLM Adaptation and Structure-Aware Attention

The model leverages semantic embeddings from OpenAI’s text-embedding-3-small to capture high-level code intent. These are concatenated with structural features derived from the code's Abstract Syntax Tree (AST). To ensure the model respects the program's logic, K-ASTRO employs a structure-aware attention bias. This mechanism uses a co-occurrence matrix from the AST to force the model to prioritize relationships between structurally related nodes, such as a variable's declaration and its subsequent use in a sensitive function call.

Diversity-Introducing AST Augmentation

To improve generalizability, K-ASTRO utilizes a novel augmentation strategy inspired by mutation testing. A library of code subtrees is constructed from the training corpus. During training, the model randomly swaps components of a function's AST with subtrees from other functions sharing the same vulnerability label. This process teaches the model to identify the core characteristics of a vulnerability regardless of minor structural variations.

The AST augmentation process extracts sub-ASTs to generate diverse training data.

Experimental Setup

The model was evaluated using three rigorous C/C++ benchmarks:

  • BigVul: Real-world CVE entries.
  • DiverseVul: A large-scale dataset designed to minimize heuristic biases.
  • PrimeVul: A high-quality dataset featuring chronological splitting and strict de-duplication to prevent data leakage.

Results and Key Insights

K-ASTRO demonstrates that inductive bias can outperform sheer scale. Despite having only 1 million parameters, it significantly outperformed GPT-4o on the BigVul CWE classification task, achieving a weighted F1 score of 76.31 compared to GPT-4o's 9.83.

Beyond accuracy, K-ASTRO is optimized for efficiency. It processes nearly 13,000 samples per second on a standard CPU and occupies only 4MB of disk space. This efficiency enables private, local code scanning without the need for expensive GPU clusters or cloud-based processing.

Conclusion

K-ASTRO represents a shift toward lightweight, specialized models for software engineering tasks. By baking structural knowledge into the architecture, the model achieves high generalizability and performance. Future work will focus on extending the model to inter-procedural analysis and expanding support to additional programming languages like Java and Python.

Summary updated

Abstract

Neural clone detection has attracted the attention of software engineering researchers and practitioners. However, most neural clone detection methods do not generalize beyond the scope of clones that appear in the training dataset. This results in poor model performance, especially in terms of model recall. In this paper, we present an Abstract Syntax Tree (AST) assisted approach for generalizable neural clone detection, or ASTRO, a framework for finding clones in codebases reflecting industry practices. We present three main components: (1) an AST-inspired representation for source code that leverages program structure and semantics, (2) a global graph representation that captures the context of an AST among a corpus of programs, and (3) a graph embedding for programs that, in combination with extant large-scale language models, improves state-of-the-art code clone detection. Our experimental results show that ASTRO improves state-of-the-art neural clone detection approaches in both recall and F-1 scores.
BibTeX
@misc{zhang2022astro,
  author = {Yifan Zhang and Junwen Yang and Haoyu Dong and Qingchen Wang and Huajie Shao and Kevin Leach and Yu Huang},
  title = {{ASTRO: An AST-Assisted Approach for Generalizable Neural Clone Detection}},
  year = {2022},
  eprint = {2208.08067},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.08067},
}
Figure