How Open Should Open Source Be?

Adam Barth, Saung Li, Benjamin I. P. Rubinstein, Dawn SongarXiv 2011 · arXiv

Abstract

Open-source transparency is often touted as a security benefit, inviting community audit. However, it also creates a "thundering herd" of public code changes that can inadvertently hide security fixes in plain sight. This paper demonstrates that attackers can leverage simple metadata and machine learning to identify unannounced security patches, potentially increasing the window of vulnerability for users by up to six times.

Introduction: The "Needle in a Haystack" Fallacy

In major open-source projects like Firefox, Linux, and Chromium, security fixes often "land" in public repositories days or weeks before an official security update reaches users. The prevailing assumption is that these patches are safe because they are buried among thousands of non-security changes—a "needle in a haystack" defense.

This research challenges that assumption by investigating whether an attacker can systematically identify these "silent" security patches. The goal is to quantify the window of vulnerability: the duration between a patch's appearance in a public repository and its installation by the end-user.

Methods of Identification

The study evaluates three levels of attack sophistication:

  1. The Bugzilla Join: This automated attack parses patch descriptions for bug IDs and queries the corresponding report in the project's bug tracker (e.g., Bugzilla). If the report is access-restricted, it is highly likely to be a security fix.
Access Denied message for a restricted bug report.
  1. Machine Learning (SVM) Ranker: To counter scenarios where bug IDs are obfuscated, the authors trained a Support Vector Machine (SVM) using non-obfuscated metadata. Key features included the patch author (e.g., security team members), modified modules (e.g., layout, js), file types, patch size, and temporal data.
  2. Random Ranker: A baseline model representing an attacker selecting patches at random.

Datasets and Ground Truth

The researchers analyzed two primary datasets from the Mozilla project:

  • Firefox 3: A 12-month window with 14,416 non-security patches, 125 security patches, and 12 security updates.
  • Firefox 3.5: A 6-month window with 7,033 patches, including 54 security-related fixes.

"Ground truth" was established by cross-referencing official Common Vulnerabilities and Exposures (CVE) lists with Mercurial commit histories.

Key Insights: Leaky Metadata and Workflow Risks

The study reveals that metadata is inherently leaky. The Author of a patch proved to be the most discriminative feature; contributions from core security team members are high-probability targets. The authors argue that obfuscating individual features is a losing game, as machine learning can aggregate multiple "weak" indicators into a strong prediction.

Distribution of security and non-security patches over time relative to release dates.

Furthermore, the "thundering herd" of daily patches is thinner than it appears. While dozens of patches land daily, an attacker only needs to audit the top one or two ranked by an ML model to find a vulnerability fix on most days. The core issue is structural: landing fixes in a public "trunk" before a release branch is ready creates an inherent window for exploitation.

Key Results

  • High Identification Rate: The Bugzilla Join method identified security patches on 260 out of 300 days of Firefox 3 development.
  • Efficiency of ML: An attacker auditing just two patches per day using the SVM ranker could gain an additional 5 months of vulnerability exposure within an 8-month period.
  • Increased Vulnerability Window: Information leaks increased the window of vulnerability by a factor of 6.4x compared to the standard 3.4-day latency for user updates.
  • Severity Targeting: The models remained highly effective at identifying "high" or "critical" severity vulnerabilities.

Limitations and Future Directions

While the paper demonstrates how to find patches, it does not quantify the effort required to reverse-engineer them into functional exploits. To mitigate these risks, the authors suggest moving security fixes to private release branches accessible only to "trusted testers." However, this introduces a trade-off: reduced community oversight may lead to more regressions. While focused on Firefox, these findings likely generalize to other major open-source ecosystems like Linux, Apache, and OpenSSL.

Summary updated

Abstract

Many open-source projects land security fixes in public repositories before shipping these patches to users. This paper presents attacks on such projects - taking Firefox as a case-study - that exploit patch metadata to efficiently search for security patches prior to shipping. Using access-restricted bug reports linked from patch descriptions, security patches can be immediately identified for 260 out of 300 days of Firefox 3 development. In response to Mozilla obfuscating descriptions, we show that machine learning can exploit metadata such as patch author to search for security patches, extending the total window of vulnerability by 5 months in an 8 month period when examining up to two patches daily. Finally we present strong evidence that further metadata obfuscation is unlikely to prevent information leaks, and we argue that open-source projects instead ought to keep security patches secret until they are ready to be released.
BibTeX
@misc{barth2011how,
  author = {Adam Barth and Saung Li and Benjamin I. P. Rubinstein and Dawn Song},
  title = {{How Open Should Open Source Be?}},
  year = {2011},
  eprint = {1109.0507},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1109.0507},
}
Figure