TL;DR
This paper presents the first large-scale empirical study on the use and effectiveness of Static Application Security Testing (SAST) tools for open-source embedded software (EMBOSS). The study found that EMBOSS projects are significantly underutilizing SAST tools, despite the critical nature of embedded systems. By applying the CodeQL SAST tool, the researchers identified hundreds of real-world defects, including security vulnerabilities, demonstrating the significant potential of SAST for improving EMBOSS quality.
What Problem Does This Paper Address?
Embedded software is ubiquitous, powering everything from medical devices to autonomous vehicles. Defects in this software can have severe consequences. Open-source embedded software (EMBOSS) is widely used, but there's a significant knowledge gap regarding its security and the effectiveness of tools used to find vulnerabilities. Existing research on SAST tools primarily focuses on traditional software, and embedded systems have unique characteristics (hardware interaction, build systems, coding practices) that might affect SAST tool performance. The paper addresses the question: "How effective are SAST tools for finding security vulnerabilities in EMBOSS, and how are they currently being used (or not used) in this domain?"
Methods
The researchers conducted a multi-phase study:
- Prevalence Study: They analyzed the Continuous Integration (CI) workflows of 258 popular EMBOSS projects and surveyed developers to understand current SAST adoption rates and developer perceptions.
- SAST Tool Selection and Configuration: They evaluated several SAST tools on a benchmark dataset and selected CodeQL as the most promising for EMBOSS due to its effectiveness. They then manually created build scripts and configured CodeQL queries to work with the diverse build systems of EMBOSS projects.
- Effectiveness Measurement: They applied the configured CodeQL tool to the EMBOSS corpus, identifying and analyzing defects. They also responsibly disclosed findings to project maintainers and proposed integrating their CodeQL workflows into project CI pipelines.
Datasets
- EMBOSS Corpus: A curated dataset of 258 popular open-source embedded software projects collected from GitHub and an RTOS index. This corpus was categorized and analyzed for SLOC and OSSF criticality scores.
- Juliet Test Suite: A labeled dataset commonly used for testing SAST tools, used for initial tool evaluation on general C/C++ code.
- Developer Survey Data: Responses from 25 developers of EMBOSS projects.
Key Insights and Ideas
- Underutilization of SAST: The vast majority of EMBOSS projects (only 4%) use sophisticated SAST tools, with most relying only on basic compiler warnings. This is a stark contrast to non-embedded OSS projects (19%).
- Developer Perceptions: Developers often perceive SAST tools as ineffective or prone to false positives, leading to low adoption. However, the study demonstrates that modern SAST tools like CodeQL can be effective.
- EMBOSS-Specific Challenges: Standard SAST tool configurations often fail on EMBOSS due to diverse build systems and unique coding patterns. Manual effort is often required to adapt these tools.
- CodeQL's Effectiveness: CodeQL proved to be the most effective SAST tool, finding hundreds of real defects, including security vulnerabilities, with a manageable false positive rate.
- Developer Acceptance of Fixes: Developers readily accepted patches for defects identified by CodeQL, indicating they value security improvements when presented with clear evidence.
- CI Integration Potential: Developers are willing to integrate SAST tools into their CI pipelines if the setup and workflow are provided and demonstrated to be effective.
Key Results
- Defects Found: CodeQL identified 709 defects across 97 (64%) of the 151 analyzed EMBOSS repositories.
- Security Vulnerabilities: Of these, 535 (75%) were security-relevant, including memory safety issues, crashes, and corruption. These were found in major projects from Microsoft, Amazon, and the Apache Foundation.
- Developer Confirmation: 376 (53%) of the identified defects were confirmed by EMBOSS engineers, primarily through accepting pull requests.
- False Positive Rate: CodeQL had an overall false positive rate of 34%, but this was polarized, with a few rules contributing most FPs. For most repositories, the FP rate was low.
- CI Integration: 37 out of 129 (29% overall, 71% of active) pull requests to integrate CodeQL workflows into project CI pipelines were merged.
- Compiler Warnings vs. SAST: Strict compiler flags (like
gcc -Wall -Wextra -Werror) detected issues in only 17% (21%) of test cases, highlighting their inadequacy for finding complex security defects compared to SAST tools.
Limitations and Future Work
- Limitations:
- Focus on Free SAST Tools: The study primarily used free tools available on GitHub Marketplace; results might differ for commercial tools.
- Focus on GitHub: Data collection was limited to GitHub projects, potentially missing EMBOSS projects on other platforms.
- Limited Developer Study: The developer survey had a small sample size, limiting generalizability.
- Construct Scope: Security vulnerabilities were limited to those detectable by the SAST tools used.
- Future Work:
- Automate the process of creating and integrating SAST workflows using Large Language Models (LLMs).
- Develop reward systems (e.g., badges) and public recognition to encourage SAST adoption.
- Create tutorials and documentation to showcase SAST effectiveness.
- Investigate more best-effort defect detection techniques tailored for EMBOSS.
Summary updated