HSVC: Transformer-based Hierarchical Distillation for Software Vulnerability Classification

Michael Fu, Van Nguyen, Chakkrit Tantithamthavorn, Trung Le, Dinh PhungOpenReview 2022 · OpenReview

Abstract

Software vulnerabilities have diverse characteristics, attacks, and impacts on software systems, stakeholders, and organizations. Such diverse characteristics of vulnerabilities (i.e., CWE-IDs) often lead to more difficulty in handling the label distributions for a Deep Learning model (e.g., addressing a highly imbalanced multi-class classification problem). However, existing vulnerability detection approaches often treat vulnerabilities equally---which does not reflect reality. In this paper, we present a new approach to solving the highly imbalanced software vulnerability classification (SVC) problem by leveraging the hierarchical structure of CWE-IDs and knowledge distillation. Specifically, we split a complex label distribution into sub-distributions based on CWE abstract types (i.e., categorizations that group similar CWE-IDs), so similar CWE-IDs can be grouped and each group will have a more balanced label distribution. We learn TextCNN teachers on each of the simplified distributions respectively, however, they only perform well in their group. Thus, we build a transformer student model to generalize the performance of TextCNN teachers through our hierarchical knowledge distillation framework. We compare our approach with source code transformer models as well as long-tailed learning approaches proposed in the vision domain. Through an extensive evaluation using the real-world 8,636 vulnerabilities, our approach outperforms all of the baselines by 1.97%-13.89%. Our framework can be applied to any transformer-based SVC such as CodeBERT, GraphCodeBERT, and CodeGPT, with slight modifications. Training code and pre-trained models are available at https://github.com/HSVC-TEAM/HSVC.
BibTeX
@misc{fu2022hsvc,
  author = {Michael Fu and Van Nguyen and Chakkrit Tantithamthavorn and Trung Le and Dinh Phung},
  title = {{HSVC: Transformer-based Hierarchical Distillation for Software Vulnerability Classification}},
  year = {2022},
  url = {https://openreview.net/forum?id=PY1wvNgwhPC},
  howpublished = {OpenReview},
}

Loading related content...

Figure