On using distributed representations of source code for the detection of C security vulnerabilities

David Coimbra, Sofia Reis, Rui Abreu, Corina S. Pasareanu, Hakan ErdogmusarXiv 2021 · OpenReview

TL;DR

This paper explores using Code2vec, a model that represents code as a collection of "path-contexts," to detect security vulnerabilities in C code. The authors found that Code2vec achieves accuracy comparable to simpler transformer models while being significantly more computationally efficient, making it a practical choice for integration into CI pipelines.

What Problem Does This Paper Address?

Software vulnerabilities are a persistent threat, and while static analysis tools aim to catch them early in the development lifecycle, they often suffer from false positives and negatives. Traditional static analysis can be cumbersome, and newer approaches using machine learning, particularly deep learning inspired by Natural Language Processing (NLP), have shown promise. However, many of these advanced models, like large transformer-based ones (e.g., BERT, CodeBERT), are computationally expensive, requiring substantial resources for training and inference. This creates a trade-off between model performance and practical usability, especially for researchers and developers with limited computational budgets. This paper investigates whether a less resource-intensive model like Code2vec can effectively detect C security vulnerabilities without sacrificing too much accuracy.

Methods

The core of this work involves adapting the Code2vec model for vulnerability detection. Code2vec represents source code functions as a "bag of path-contexts." These path-contexts are derived from the Abstract Syntax Tree (AST) of the code, capturing syntactic relationships between different code elements. Specifically, a path-context is a triplet consisting of the start terminal, the AST path connecting it to the end terminal, and the end terminal.

The process involved:

  1. Code Extraction: Using the astminer library to extract AST path-contexts from C functions.
  2. Code2vec Training: Training Code2vec on these path-contexts to classify functions as either "vulnerable" or "non-vulnerable" (represented as "vuln" and "safe" tokens).
  3. Evaluation: Assessing the model's performance using metrics like accuracy, precision, recall, and F1-score on the Devign dataset, as part of the CodeXGLUE benchmark.

The authors also compared Code2vec's computational requirements (training time and memory) against transformer-based models.

Datasets

The primary dataset used in this study is Devign.

  • Devign: This is a publicly available dataset containing 27,318 manually labeled C functions, collected from the QEMU and FFmpeg projects. The labels indicate whether a function is vulnerable or non-vulnerable. The dataset was split into training, validation, and testing sets with an 80%/10%/10% distribution, respectively, as prepared by the CodeXGLUE benchmark.

Key Insights and Ideas

  • Code2vec's Viability for Vulnerability Detection: The study demonstrates that Code2vec, a model not originally designed for vulnerability detection, can perform comparably to simpler transformer models on this task. This suggests that its approach of capturing code semantics through path-contexts is effective.
  • The Efficiency Advantage: The most significant insight is Code2vec's dramatically lower computational footprint. It trains much faster and requires significantly less memory than large transformer models, making it a practical option for resource-constrained environments or for integration into continuous integration (CI) pipelines where speed is critical.
  • Trade-off Between Expressiveness and Efficiency: The paper highlights the inherent trade-off between highly expressive, pre-trained transformer models and simpler, more efficient models like Code2vec. While transformers might achieve higher accuracy by leveraging vast pre-training data, Code2vec offers a compelling balance for practical applications.

Key Results

  • Accuracy: Code2vec achieved an accuracy of 61.43% on the Devign dataset. This performance is comparable to pre-trained RoBERTa (61.05%) and outperforms naive NLP-based methods like BiLSTM (59.37%) and TextCNN (60.69%).
  • Computational Efficiency:
    • Training Time: Code2vec completed a full training session (20 epochs) in approximately 5 minutes on a consumer-grade GPU. In contrast, CodeBERT required about 7 hours for a 5-epoch session on similar hardware.
  • Memory Consumption: A single training step for Code2vec used about 600MB of GPU memory, while CodeBERT required approximately 2.5GB.

Limitations and Future Work

  • Limitations:
    • Code2vec was outperformed by more advanced models like PLBART, C-BERT, and CodeBERT, which leverage transformer architectures and extensive pre-training on code data. This indicates that while Code2vec is efficient, it may not capture the full semantic richness of code as effectively as these larger models.
  • The astminer library was unable to extract path-contexts from 71 functions in the Devign dataset, leading to their exclusion from the study.
  • Future Work:
    • The authors suggest exploring hyperparameter tuning for Code2vec, such as optimizing maximum path-context length and width, learning rate, and batch size, to potentially improve its performance.
  • They also note that Code2vec remains an attractive choice for developers for code intelligence tasks due to its comparable performance and ease of integration into CI pipelines compared to static analyzers or larger models.

Summary updated

Abstract

This paper presents an evaluation of the code representation model Code2vec when trained on the task of detecting security vulnerabilities in C source code. We leverage the open-source library astminer to extract path-contexts from the abstract syntax trees of a corpus of labeled C functions. Code2vec is trained on the resulting path-contexts with the task of classifying a function as vulnerable or non-vulnerable. Using the CodeXGLUE benchmark, we show that the accuracy of Code2vec for this task is comparable to simple transformer-based methods such as pre-trained RoBERTa, and outperforms more naive NLP-based methods. We achieved an accuracy of 61.43% while maintaining low computational requirements relative to larger models.
BibTeX
@misc{coimbra2021using,
  author = {David Coimbra and Sofia Reis and Rui Abreu and Corina S. Pasareanu and Hakan Erdogmus},
  title = {{On using distributed representations of source code for the detection of C security vulnerabilities}},
  year = {2021},
  eprint = {2106.01367},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.01367},
}
Figure