Automatically assessing vulnerabilities discovered by compositional analysis

Saahil Ognawala, Ricardo Nales Amato, Alexander Pretschner, Pooja Kulkarni

TL;DR

This paper introduces a novel framework that automatically assigns severity scores (using CVSS3) to software vulnerabilities discovered by compositional symbolic execution. By analyzing features of vulnerable code and its context, the system helps developers prioritize which bugs to fix first, improving upon existing tools that often lack this crucial prioritization information.

What Problem Does This Paper Address?

Finding vulnerabilities in complex, large-scale software is a significant challenge. While automated testing methods like symbolic execution are effective, they can produce a large number of potential vulnerabilities. Existing tools often report these findings without a clear indication of their severity, leaving developers to manually triage and prioritize them. This is a time-consuming and subjective process. Furthermore, compositional analysis, which breaks down large programs into smaller, manageable components, can find more vulnerabilities but still lacks a robust severity assessment mechanism. This paper aims to bridge this gap by providing an automated way to assess the severity of vulnerabilities identified through compositional analysis.

Methods

The core of this work involves a machine learning approach to predict CVSS3 base scores. The process begins with data collection from the National Vulnerability Database (NVD) for C programs, specifically focusing on buffer-overflow vulnerabilities with CVSS3 scores and identified vulnerable functions.

Next, the compositional analysis is performed using a tool called Macke. Macke symbolically executes individual program components (functions) and then analyzes how these components interact to identify vulnerabilities and their reachability.

The crucial step is feature extraction. For each vulnerable function identified by Macke, several features are extracted from the program's call graph and analysis results. These include:

  • Node degree (in and out): Number of callers and callees.
  • Distance to interface: Shortest path from a program entry point.
  • Clustering coefficient: How interconnected a function's neighbors are.
  • Node path length: Average distance to reachable nodes.
  • Vulnerabilities discovered: Number of unique vulnerable instructions found by Macke.
  • Maximum length of infection: The longest chain of function calls through which a vulnerability can be exploited.

These features are then used to train machine learning models (Random Forest and Naive Bayes classifiers) to predict individual CVSS3 base score metrics. The models are trained on a dataset of known vulnerabilities and their scores, and then evaluated on their ability to predict scores for previously unseen vulnerabilities.

Finally, the framework includes an interactive reporting interface and a feedback gathering mechanism. Developers and security experts can interact with the call graph, view predicted scores, and provide feedback. This feedback is used to refine the models by adding new features, such as function size, approximate function complexity (basic blocks), and the number of pointer parameters.

Datasets

The primary dataset used for training and evaluation is derived from the National Vulnerability Database (NVD). The authors filtered NVD data to include:

  • Vulnerabilities following CVSS3 notation.
  • Reports pertaining to C programs.
  • Buffer-overflow vulnerabilities.
  • Reports that explicitly name the vulnerable function.

They also supplemented this data by manually scoring additional vulnerable functions discovered by Macke, as the NVD data alone was insufficient for training. The paper analyzes 21 open-source programs.

Key Insights and Ideas

  • Context is King for Severity: The paper strongly suggests that understanding the context of a vulnerability – its interaction with other components, its complexity, and its potential for exploitation – is crucial for accurately assessing its severity. This goes beyond simply identifying the presence of a bug.
  • Compositional Analysis Provides Rich Features: Macke's compositional analysis, by focusing on inter-component interactions and reachability, generates features (like "maximum length of infection") that are highly correlated with CVSS3 scores, proving more effective than purely syntactic features.
  • Developer Feedback is Invaluable: The iterative process of gathering feedback from security experts and incorporating it into feature engineering significantly improved the accuracy of the prediction models. This highlights the importance of human expertise in refining automated tools.
  • Not All CVSS Metrics are Equally Predictable: Some CVSS metrics, like Attack Vector and Integrity Impact, are harder to predict solely from code features, likely due to their reliance on external deployment and usage context. Others, like Attack Complexity and Availability Impact, are more amenable to prediction from code analysis.

Key Results

  • The initial models, using features extracted from compositional analysis, achieved notable accuracy in predicting CVSS3 base scores. For instance, Privileges Required (PR) and Scope (S) were predicted with 91% accuracy by Random Forest, and User Interaction (UI) with 73%.
  • After incorporating expert feedback and adding new features (function size, complexity, pointer parameters), the accuracy of the prediction models improved significantly. Privileges Required (PR) and Scope (S) were predicted with 100% accuracy by both Random Forest and Naive Bayes, and User Interaction (UI) with 95% accuracy.
  • Macke discovered 296 unique vulnerable functions across the 21 analyzed programs, which was more than the number reported in NVD, indicating its effectiveness in finding vulnerabilities.
  • Qualitative feedback from experts indicated that the tool was generally useful, though they expressed a desire for more precise vulnerability pinpointing (specific lines of code) and noted the impact of function size on analysis.

Limitations and Future Work

  • Subjectivity of CVSS: The authors acknowledge that CVSS scores themselves can be subjective and may not always perfectly reflect real-world bug-fixing priorities.
  • External Threats to Validity: The results are based on open-source programs maintained by large communities. Generalizing these findings to proprietary software, embedded systems, or different programming languages might require further investigation.
  • Predictability of Certain Metrics: Attack Vector and Integrity Impact remain challenging to predict accurately using only code-based features, suggesting the need for incorporating external information.
  • Future Work: The authors suggest exploring other vulnerability discovery tools beyond Macke to assess the adaptability of their feature set. They also propose incorporating external information sources (like function requirements or deployment context) to improve the prediction of metrics that are currently difficult to assess. Additionally, pinpointing the exact vulnerable instruction, rather than just the function, is identified as a desirable enhancement.

Summary updated

Abstract

Testing is the most widely employed method to find vulnerabilities in real-world software programs. Compositional analysis, based on symbolic execution, is an automated testing method to find vulnerabilities in medium- to large-scale programs consisting of many interacting components. However, existing compositional analysis frameworks do not assess the severity of reported vulnerabilities. In this paper, we present a framework to analyze vulnerabilities discovered by an existing compositional analysis tool and assign CVSS3 (Common Vulnerability Scoring System v3.0) scores to them, based on various heuristics such as interaction with related components, ease of reachability, complexity of design and likelihood of accepting unsanitized input. By analyzing vulnerabilities reported with CVSS3 scores in the past, we train simple machine learning models. By presenting our interactive framework to developers of popular open-source software and other security experts, we gather feedback on our trained models and further improve the features to increase the accuracy of our predictions. By providing qualitative (based on community feedback) and quantitative (based on prediction accuracy) evidence from 21 open-source programs, we show that our severity prediction framework can effectively assist developers with assessing vulnerabilities.
BibTeX
@inproceedings{ognawala2018automatically,
  author = {Saahil Ognawala and Ricardo Nales Amato and Alexander Pretschner and Pooja Kulkarni},
  title = {{Automatically assessing vulnerabilities discovered by compositional analysis}},
  booktitle = {MASES@ASE},
  year = {2018},
  doi = {10.1145/3243127.3243130},
}
Figure