An Empirical Study of Vulnerability Rewards Programs

Matthew Finifter, Devdatta Akhawe, David A. WagnerUSENIX Security 2013 · OpenReview

TL;DR

This paper empirically analyzes the Google Chrome and Mozilla Firefox vulnerability rewards programs (VRPs) to understand their effectiveness and cost-efficiency. It finds that VRPs are a cost-effective way to discover vulnerabilities, with Chrome's tiered reward structure and higher profile proving more successful in attracting external contributors than Firefox's fixed payout model.

What Problem Does This Paper Address?

Software vendors are increasingly adopting Vulnerability Rewards Programs (VRPs), also known as bug bounty programs, to incentivize security researchers to find and responsibly disclose software vulnerabilities. However, there's an ongoing debate about the true value and effectiveness of these programs. Some vendors, like Microsoft, have questioned their return on investment, and it's unclear if the rewards offered are sufficient to attract researchers compared to the lucrative underground market for vulnerabilities. This paper aims to provide empirical evidence to guide vendors in launching and evolving their VRPs by analyzing two mature, high-profile programs: Google Chrome and Mozilla Firefox.

Methods

The authors conducted a quantitative analysis of vulnerability data from the Google Chrome and Mozilla Firefox VRPs over a three-year period. They collected data on:

  • Vulnerabilities reported and rewarded: This included identifying bugs found through the VRP and those affecting stable releases.
  • Severity of vulnerabilities: Categorized as low, medium, high, and critical.
  • Reward amounts: Documenting the monetary payouts for discovered vulnerabilities.
  • Reporter information: Distinguishing between internal (employee) and external (independent) researchers.
  • Time to patch: Measuring the duration between a vulnerability report and the release of a patch.
  • Independent discoveries: Identifying instances where a vulnerability was found by multiple parties.

They employed a heuristic to distinguish between internal and external bug reports, assuming that rewarded bugs were reported by external researchers. For time-to-patch analysis, they focused on the time to release a patch rather than just committing it, as this is more relevant to end-users.

Datasets

The study utilized two primary datasets:

  • Google Chrome VRP Data: Collected from the official Chromium bug tracker, including bugs marked with "Reward" and "Security-Impact: Stable" labels. This yielded 501 VRP bugs and 1347 stable release bugs.
  • Mozilla Firefox VRP Data: Collected from Bugzilla and Mozilla Foundation Security Advisories (MFSA). This provided 190 VRP bugs and 613 stable release bugs.

The authors also made their combined dataset publicly available for independent analysis.

Key Insights and Ideas

  • VRPs are cost-effective: The study strongly suggests that VRPs are a more cost-effective way to find vulnerabilities than hiring full-time security researchers, potentially being 2-100 times more efficient.
  • Chrome's tiered rewards are more engaging: Chrome's structure, offering a range of rewards with high potential payouts, appears to be more effective at encouraging community participation than Firefox's fixed $3,000 payout for critical/high vulnerabilities.
  • Privilege separation matters for severity: Chrome's privilege-separated architecture leads to a higher proportion of its critical vulnerabilities being classified as high severity in Firefox, highlighting the impact of architectural design on vulnerability classification.
  • Repeat contributors are key: Both VRPs benefit from repeat contributors, but Chrome has seen a more consistent reliance on them, suggesting a well-established ecosystem.
  • Time-to-patch is crucial for responsible disclosure: Firefox's higher variance in patching critical vulnerabilities could deter responsible disclosure, as vendors need to respond promptly to maintain researcher trust.
  • High maximum rewards can attract participation: Similar to lotteries, high potential rewards (like Chrome's $10,000+) can incentivize participation even if the average payout is low.

Key Results

  • Bug Discovery: Chrome VRP uncovered approximately 2.6 times more vulnerabilities than Firefox (501 vs. 190).
  • VRP Contribution: VRPs accounted for 27.5% of Chrome's patched vulnerabilities and 24.1% of Firefox's.
  • Cost: Over three years, both Chrome and Firefox spent a similar amount on rewards, around $570,000-$580,000.
  • Average Daily Cost: Chrome's VRP cost approximately $485/day, while Firefox's cost $658/day.
  • Vulnerability Severity: Firefox had a much higher ratio of critical to high vulnerabilities than Chrome, likely due to the lack of privilege separation.
  • Time to Patch: Chrome generally patched vulnerabilities faster and with less variance, especially for critical ones, compared to Firefox.
  • External Contributors: Chrome saw significantly higher external participation, with external contributions nearly matching internal ones, while Firefox lagged behind.
  • Individual Earnings: While contributing to a single VRP might not be a full-time job, highly successful researchers could earn substantial amounts, especially if contributing to multiple VRPs.

Limitations and Future Work

The authors acknowledge several limitations:

  • Data Completeness: The datasets might be incomplete due to manual labeling processes and reliance on specific bug tracker labels.
  • Comparability of Skills: The study assumes comparable skills and resources are needed to find vulnerabilities in both browsers, which might not be entirely true.
  • Firefox Reward Data: Precise reward amounts for all Firefox bugs were not publicly available, leading to potential minor inaccuracies in cost analysis.
  • Motivation of Researchers: The study did not deeply explore the motivations of security researchers beyond financial incentives.

Future work suggested includes:

  • Economic Modeling: Developing economic models for VRPs.
  • VRP Lifecycle Analysis: Identifying typical patterns, trajectories, and phases of VRPs.
  • Studying Failed VRPs: Analyzing unsuccessful VRPs to understand potential pitfalls.
  • Broader VRP Analysis: Gathering and analyzing data from more VRPs to paint a more complete picture.
  • Confirming Hypotheses: Further research is encouraged to confirm hypotheses regarding zero-day decay rates and the success of independent researchers.

Summary updated

BibTeX
@inproceedings{finifter2013empirical,
  author = {Matthew Finifter and Devdatta Akhawe and David A. Wagner},
  title = {{An Empirical Study of Vulnerability Rewards Programs}},
  booktitle = {USENIX Security},
  year = {2013},
  url = {https://openreview.net/forum?id=ouWYg63wjq},
}
Figure