Data correction and evolution analysis of the ProgrammableWeb service ecosystem

Mingyi Liu, Zhiying Tu, Yeqi Zhu, Xiaofei Xu, Zhongjie Wang, Quan Z. Sheng

TL;DR

This paper provides a much-needed "reality check" for the Web service research community by uncovering significant data quality issues in the industry-standard ProgrammableWeb dataset. By developing a novel probabilistic method to correct service "death times" and active statuses, the authors reveal that the API ecosystem is actually in a state of decline rather than the "explosive growth" reported by previous studies.

What Problem Does This Paper Address?

For over a decade, researchers have used the ProgrammableWeb (PW) dataset to study the "API economy"—how web services (APIs) are composed into applications (mashups). However, most existing evolution analyses share a fatal flaw: they assume the data is high-quality and ignore service obsolescence.

In reality, many APIs listed as "active" are dead links, and the "death dates" provided by PW are often administrative artifacts (e.g., thousands of APIs being marked "dead" on a single day in April 2014) rather than actual timestamps of service failure. This leads to a "blindly optimistic" view of the ecosystem's health, resulting in inaccurate service recommendations and flawed models of how these networks grow.

Methods

The authors tackle this through a multi-stage pipeline of identification, correction, and re-analysis:

  1. Data Auditing: They used automated network request testing (checking if API endpoints actually respond) and manual inspection by five expert coders to identify three main issues: untrustworthy death times, incorrect API statuses, and incorrect mashup compositions.
  2. Probabilistic Correction: Since the exact death date for many services is unknown, the authors estimated service lifespans using a Normal Distribution-based method. They validated this distribution against a manually verified sample of 200 APIs using Z-statistics to ensure the corrected "deathpool" reflected a realistic overall distribution.
  3. Dynamic Network Modeling: They modeled the ecosystem at three levels:
  4. Mashup-API (M-A): A bipartite graph showing which mashups invoke which APIs.
  5. API-API (A-A): A network where APIs are linked if they are used together in the same mashup.
  6. Category-Category (C-C): A high-level view of how different service domains (e.g., Social, Mapping, Search) interact.

Datasets

  • ProgrammableWeb (Original): The raw dataset containing 23,881 APIs and 7,973 mashups collected as of January 2021.
  • Corrected ProgrammableWeb Dataset: The authors have released their cleaned version, which includes corrected active statuses and estimated active timeframes, on GitHub.

Key Insights and Ideas

  • The "Deathpool" is a Mess: The authors found that 85.2% of unavailable APIs were incorrectly marked as "active" in the original dataset. Many APIs were "born" after they were supposedly "dead" due to administrative errors in the PW database.
  • The Ecosystem is Shrinking: Contrary to the narrative of "explosive growth," the number of active APIs and mashups actually peaked around 2014 and has been declining since. The authors suggest this might be due to the shift from Web-based mashups to mobile apps.
  • Complexity has Plateaued: Since 2012, the complexity of mashups (the number of APIs they invoke) has not increased. Most mashups remain simple, typically using only 2-3 APIs.
  • Popularity $\neq$ Survival: Interestingly, the study found that using frequently co-occurring API pairs (like Twitter + Google Maps) does not actually increase a mashup's chance of survival. In fact, some high-frequency pairs have survival rates as low as 20%.

Key Results

  • Active Status: Only 44.7% of the APIs listed in the original dataset are truly active.
  • Network Laws: While previous studies claimed the API network follows a strict Power Law distribution (where a few "super-nodes" dominate), the authors' statistical $p$-value analysis suggests this hypothesis is less stable than previously thought.
  • Diversity Decline: The diversity of the ecosystem (measured by the number of active service categories) entered a plateau and began to decline rapidly after 2017.
  • Connectivity: The size of the "largest connected component" in the API network has continued to shrink since 2013, indicating a more fragmented and less collaborative ecosystem.

Limitations and Future Work

The authors acknowledge that their correction method is probabilistic. While it accurately restores the overall distribution of the ecosystem, it cannot guarantee that the estimated death date for a specific individual API is 100% correct.

Future work could involve building more sophisticated regression models to predict service lifespans if more labeled data becomes available. Additionally, the authors suggest that the research community needs to look beyond ProgrammableWeb to capture the modern, more fragmented landscape of mobile and enterprise APIs that may not be indexed on public repositories.

Summary updated

Abstract

The evolution analysis on Web service ecosystems has become a critical problem as the frequency of service changes on the Internet increases rapidly. Developers need to understand these evolution patterns to assist in their decision-making on service selection. ProgrammableWeb is a popular Web service ecosystem on which several evolution analyses have been conducted in the literature. However, the existing studies have ignored the quality issues of the ProgrammableWeb dataset and the issue of service obsolescence. In this study, we first report the quality issues identified in the ProgrammableWeb dataset from our empirical study. Then, we propose a novel method to correct the relevant evolution analysis data by estimating the life cycle of application programming interfaces (APIs) and mashups. We also reveal how to use three different dynamic network models in the service ecosystem evolution analysis based on the corrected ProgrammableWeb dataset. Our experimental experience iterates the quality issues of the original ProgrammableWeb and highlights several research opportunities.
BibTeX
@article{liu2021data,
  author = {Mingyi Liu and Zhiying Tu and Yeqi Zhu and Xiaofei Xu and Zhongjie Wang and Quan Z. Sheng},
  title = {{Data correction and evolution analysis of the ProgrammableWeb service ecosystem}},
  journal = {Journal of Systems and Software},
  year = {2021},
  url = {https://openreview.net/forum?id=ATanVB1J6T},
}

Loading related content...

Figure