Deploying Search Based Software Engineering with Sapienz at Facebook

Nadia Alshahwan, Xinbo Gao, Mark Harman, Yue Jia, Ke Mao, Alexander Mols, Taijin Tei, И. В. ЗоринLecture Notes in Computer Science 2018 · DOI · OpenAlex

Abstract

We describe the deployment of the Sapienz Search Based Software Engineering (SBSE) testing system. Sapienz has been deployed in production at Facebook since September 2017 to design test cases, localise and triage crashes to developers and to monitor their fixes. Since then, running in fully continuous integration within Facebook’s production development process, Sapienz has been testing Facebook’s Android app, which consists of millions of lines of code and is used daily by hundreds of millions of people around the globe. We continue to build on the Sapienz infrastructure, extending it to provide other software engineering services, applying it to other apps and platforms, and hope this will yield further industrial interest in and uptake of SBSE (and hybridisations of SBSE) as a result.

Introduction: The Challenge of Scale

Testing mobile applications at the scale of Facebook presents a monumental engineering challenge. With codebases spanning millions of lines and a development velocity exceeding 100,000 code commits (diffs) per week, traditional manual testing is insufficient. While automated unit testing is a standard industry practice, it often fails to capture system-level regressions and can lead to "false positives"—reports of bugs that are not actually reachable by users. Such noise erodes developer trust and slows down the release cycle.

To address this, Facebook deployed Sapienz, a production-scale testing system that utilizes multi-objective evolutionary algorithms to automatically find, triage, and verify crashes. By focusing on system-level testing via the Graphical User Interface (GUI), Sapienz ensures that every crash it reports is a "real" failure experienced by the end-user.

The Sapienz Architecture and Workflow

Sapienz is built on the principles of Multi-objective Search-Based Software Engineering (SBSE). It employs the NSGA-II evolutionary algorithm to optimize test sequences for three simultaneous goals: maximizing code coverage, maximizing the number of unique crashes found, and minimizing the length of the test sequence. Shorter sequences are prioritized because they are significantly easier for developers to debug.

The three primary workflows of Sapienz: Evolution, Crash Reproduction, and Fix Detection.

The system operates through three distinct, interconnected workflows:

  1. Evolution Workflow (Hourly): This is the core search engine. It generates new test sequences by "mutating" and "crossing over" successful previous tests to explore new areas of the application.
  2. Crash Reproduction Workflow (Hourly): When a crash is detected, this workflow attempts to reproduce it reliably, extracting stack traces and crash data to provide a "debug payload" for developers.
  3. Fix Detection Workflow (Daily): This workflow monitors developer tasks to verify if a reported bug has truly been resolved. It uses a sophisticated state machine with "cooling" and "resurrection" windows to account for test flakiness, ensuring a bug is only marked as fixed if it remains absent over a sustained period.

Integration into the Production Pipeline

Sapienz is not a standalone tool but a deeply integrated part of Facebook’s Continuous Integration (CI) environment. It interacts directly with Phabricator (the code review tool) and the continuous build system to provide rapid feedback to engineers.

Sapienz integration within the Facebook developer ecosystem, showing the interaction between engineers, Phabricator, and the build pipeline.

As shown in the integration model, Sapienz runs as a recurring operation every hour, fetching builds and feeding crash localization data back into the Phabricator system. This allows reviewers and engineers to see Sapienz reports alongside their code changes. A key innovation in this pipeline is the ASER (Automated Scientific Experimental Reporting) framework. ASER uses inferential statistics to evaluate new versions of the Sapienz algorithm in a tiered approach (Lite, Standard, and ProdExp), ensuring that only improvements that provide a faster or more accurate signal reach the production environment.

Key Insights and Results

Since its deployment, Sapienz has achieved a remarkable 75% fix rate for the crashes it reports. This high success rate is attributed to several factors:

  • Existence Proofs: Sapienz provides developers with a video of the crash and a stack trace, turning a vague report into an undeniable proof of failure.
  • Static and Dynamic Synergy: When Sapienz is used to verify potential bugs found by Infer (Facebook’s static analysis tool), the fix rate approaches 100%. Static analysis identifies potential vulnerabilities, while Sapienz proves they are reachable and exploitable.
  • Prioritizing Signal over Coverage: Production experience revealed that "Time-to-Signal" is more critical than absolute coverage. Developers value a fast, reliable report over a comprehensive but slow one.

Future Directions

The Sapienz team is currently working on moving beyond "implicit oracles" (crashes) toward "automated oracles" capable of detecting logic errors and UI glitches. Additionally, research is underway to incorporate "social state"—the complex web of user data and connections—into the search space. The ultimate vision is to close the loop entirely through Genetic Improvement (GI), where the system not only finds and triages bugs but also suggests automated code fixes for developer approval.

Summary updated

Abstract

We describe the deployment of the Sapienz Search Based Software Engineering (SBSE) testing system. Sapienz has been deployed in production at Facebook since September 2017 to design test cases, localise and triage crashes to developers and to monitor their fixes. Since then, running in fully continuous integration within Facebook’s production development process, Sapienz has been testing Facebook’s Android app, which consists of millions of lines of code and is used daily by hundreds of millions of people around the globe. We continue to build on the Sapienz infrastructure, extending it to provide other software engineering services, applying it to other apps and platforms, and hope this will yield further industrial interest in and uptake of SBSE (and hybridisations of SBSE) as a result.
BibTeX
@inproceedings{alshahwan2018deploying,
  author = {Nadia Alshahwan and Xinbo Gao and Mark Harman and Yue Jia and Ke Mao and Alexander Mols and Taijin Tei and И. В. Зорин},
  title = {{Deploying Search Based Software Engineering with Sapienz at Facebook}},
  booktitle = {Lecture Notes in Computer Science},
  year = {2018},
  doi = {10.1007/978-3-319-99241-9\_1},
}
Figure