TL;DR
This paper introduces a novel empirical model to quantify the economic value of static analysis tools, addressing the common challenge of demonstrating their ROI. By applying this model to Microsoft's Static Driver Verifier (SDV), the authors reveal that while these tools offer significant realized value, their potential value (cost avoided by fixing all found defects) far exceeds this, highlighting a critical area for improvement in development processes.
What Problem Does This Paper Address?
Static analysis tools promise to catch bugs early in the software development lifecycle, potentially saving significant costs associated with fixing defects later. However, despite their technical capabilities, these tools often struggle with adoption and integration into development workflows. A key reason for this is the difficulty in quantifying their economic impact. Developers and managers need concrete numbers to justify the investment in these tools, understand their true return on investment (ROI), and prioritize their use. This paper aims to bridge this gap by providing a framework to measure the economic value of static analysis tools.
Methods
The authors propose a concise empirical model to evaluate static analysis tools. The core of the model revolves around calculating the "cost of a defect" and the "value of a tool."
- Cost of a Defect (Cdefect): This is calculated by considering two main components:
- Defect Removal Cost (Crem): The cost to fix the defect after the product has been shipped.
- Cost of Occurrence (Coccur * F * T): The cost incurred when a defect manifests in the field, multiplied by its frequency (F) and the total number of product installations (T).
- The model also considers the "Resolution Status" (whether the defect was fixed before shipping or not) and "Severity" of the defect.
- Value of a Tool (Vtool): This is defined as the cost avoided by shipping products with fewer defects. It's split into:
- Realized Value (Vrealized): The value derived from defects that were found and fixed by the tool before shipping.
- Potential Value (Vpotential): The value that could be realized if all defects found by the tool were fixed, even those not fixed during the initial development cycle.
- Cost of a Tool (Ctool): This includes both the research and development (R&D) costs and the application costs (integrating and running the tool systematically).
Datasets
The paper uses data from the Windows 7 timeframe for the Static Driver Verifier (SDV) tool. This dataset includes:
- A breakdown of defects discovered by SDV, categorized by type (code, documentation).
- Severity levels for code defects.
- Resolution status (fixed vs. not fixed) for code defects.
The authors note that this data is specific to internal defect reporting and may not capture all defects discovered by developers using SDV for their own validation.
Key Insights and Ideas
- The "Value Gap": The most striking insight is the significant difference between a tool's realized value (what's currently being saved) and its potential value (what could be saved if all identified defects were addressed). This gap represents a major opportunity for improvement.
- Beyond Tangible Value: The model primarily focuses on quantifiable economic costs and benefits. However, the authors acknowledge a crucial "intangible value" that static analysis tools provide, such as forcing better code reviews, prompting design re-evaluations, and improving the overall ecosystem quality. This intangible value is not captured by the current model but is recognized as significant.
- Application Cost is Crucial: The paper emphasizes that the cost of applying a static analysis tool (integrating it, running it systematically, and automating its use) is often overlooked but is critical for realizing its value. Neglecting this can lead to tools having a shorter lifespan and much less realized value.
- Severity Matters: The model explicitly incorporates defect severity, recognizing that higher-severity defects have a disproportionately larger impact on cost and value.
Key Results
Applying the model to SDV during the Windows 7 timeframe yielded the following:
- Total Value of SDV: Approximately $16.6 million.
- Realized Value: $3.6 million. This is the value currently being captured by fixing defects found by SDV.
- Potential Value: $13 million. This represents the additional value that could be unlocked if all defects identified by SDV were fixed.
- The paper highlights that the potential value far exceeds the realized value, indicating that development organizations are not fully leveraging the benefits of tools like SDV.
Limitations and Future Work
- Domain Specificity: The model was applied to a single tool (SDV) within a specific domain (Windows kernel drivers). The authors acknowledge that the model's generalizability to other domains and tools needs further investigation.
- Assumptions: The model relies on assumptions for defect frequency and cost of occurrence, which may vary across different contexts.
- Intangible Value: The current model does not quantitatively capture the significant intangible benefits of static analysis tools.
Future work suggested includes:
- Applying the model to a wider range of tools and domains.
- Extending the model to better incorporate intangible values.
- Developing processes to help organizations better realize the potential value of static analysis tools.
Summary updated