Quantifying dynamics of failure across science, startups, and security

Yian Yin, Yang Wang, James A. Evans, Dashun WangarXiv 2019 · arXiv

Introduction

In the popular narrative of innovation, failure is often framed as a necessary stepping stone to success. We are frequently reminded that Thomas Edison failed a thousand times before perfecting the lightbulb, or that Henry Ford’s early ventures ended in bankruptcy. However, this narrative suffers from survivorship bias. For every Edison, there are countless individuals and organizations that fail repeatedly without ever reaching a breakthrough.

What distinguishes a "successful" failure from one that leads to permanent stagnation? In "Quantifying dynamics of failure across science, startups, and security," researchers move beyond anecdotal evidence to develop a mathematical framework that explains why some agents learn from their mistakes while others remain trapped in a cycle of futility.

The $k$-Model: A Framework for Learning

To understand these dynamics, the authors proposed a one-parameter model, the $k$-model, which describes how agents build upon their past attempts. In this model, every attempt is viewed as a collection of components. When initiating a new attempt, an agent has two options for each component:

  1. Create: Develop a new version from scratch (with probability $p$).
  2. Reuse: Select the best version from the previous $k$ attempts (with probability $1-p$).

The parameter $k$ represents the agent’s "memory" or the depth of experience they effectively utilize. Using tools from statistical physics, the researchers identified a critical threshold, $k^*$, that marks a phase transition in performance.

  • The Stagnation Regime ($k < k^*$): Agents fail to effectively retain or build upon high-quality components. They essentially "thrash around," making changes that do not lead to cumulative improvement. In this regime, the time between failures remains constant, and the quality of work does not improve over time.
  • The Progression Regime ($k > k^*$): Agents systematically refine their work by leveraging past successes. This leads to a power-law increase in efficiency and quality. In this regime, the time between successive attempts decreases, indicating that the agent is learning and iterating faster.

Empirical Evidence Across Disparate Domains

The researchers tested the $k$-model against three massive datasets representing different types of human endeavor:

  • Science ($D_1$): 776,721 grant applications submitted to the National Institutes of Health (NIH) between 1985 and 2015.
  • Startups ($D_2$): Investment records for 58,111 companies from the VentureXpert database (1970–2016).
  • Security ($D_3$): 170,350 terrorist attacks by 3,178 organizations from the Global Terrorism Database (1970–2017).

Across all three domains, the researchers tracked "failure streaks"—sequences of attempts that ended in either a defined success (e.g., a funded grant, an IPO, or a fatal attack) or permanent dropout from the system.

Key Insights: Timing as a Signal

One of the most profound findings of the study is that the timing between attempts is a more reliable predictor of ultimate success than the apparent quality of the attempts themselves.

In the progression regime, the inter-event time follows a power-law decay; agents literally get faster at failing until they succeed. Conversely, in the stagnation regime, the time between attempts does not decrease. This suggests that the "speed" of iteration is a statistical signature of effective learning.

Furthermore, the authors derived a scaling identity, $\beta + \gamma = 1$, where $\beta$ represents the distribution of failure streaks and $\gamma$ represents the temporal scaling exponent. This identity mathematically links the duration of a failure streak to the rate at which an agent learns.

Early Predictability and the Stagnation Trap

The model reveals that successful and unsuccessful agents often look identical at the start of their journeys. Their initial performance metrics and demographic characteristics are frequently indistinguishable. Their paths only diverge based on how they process failure.

Remarkably, the researchers found that by analyzing the timing of just the first three failures, they could predict ultimate success with an AUC (Area Under the Curve) of nearly 0.7. This predictive power exists regardless of traditional factors like prestige, funding history, or institutional support.

In the "Success" group, the penultimate attempt was statistically superior to the first. In the "Non-success" group, there was zero statistical improvement between the first and last attempts, even after a long sequence of failures. This confirms that simply "trying again" is insufficient; without crossing the $k^*$ threshold, repeated attempts do not lead to progress.

Conclusion

The research suggests that success is not merely a matter of persistence or luck, but a result of how effectively one incorporates past experience. The existence of a phase transition implies that small changes in how an individual or organization manages its "memory" of failure can lead to radically different outcomes. While the model has limitations—such as assuming components are independent and ignoring vicarious learning—it provides a powerful new lens through which to view the mechanics of achievement. To succeed, one must not only fail often but fail faster and smarter, ensuring that each attempt is a genuine refinement of the last.

Summary updated

BibTeX
@misc{yin2019quantifying,
  author = {Yian Yin and Yang Wang and James A. Evans and Dashun Wang},
  title = {{Quantifying dynamics of failure across science, startups, and security}},
  year = {2019},
  eprint = {1903.07562},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.07562},
}
Figure