Empirical Research on the Evaluation Model and Method of Sustainability of the Open Source Ecosystem

Zhifang Liao, Libing Deng, Xiaoping Fan, Yan Zhang, Hui Liu, Xiaofei Qi, Yun ZhouSymmetry 2018 · DOI

TL;DR

This paper introduces a novel evaluation framework and model to quantitatively assess the sustainability of Open Source Ecosystems (OSEs), addressing the lack of universal metrics and methodologies. By applying this framework to the Stack Overflow (SO) ecosystem, they demonstrate its effectiveness in analyzing sustainability trends and predicting future development.

What Problem Does This Paper Address?

The rapid growth of open-source software (OSS) has led to the emergence of complex "Open Source Ecosystems" (OSEs). While the sustainability of these ecosystems is crucial for their long-term viability and for stakeholders like investors and developers, there's a significant gap in how to objectively measure it. Existing research often focuses on individual projects rather than the entire ecosystem, and lacks a standardized, universal model and methodology for assessing sustainability. This makes it difficult to compare OSEs, identify weaknesses, and make informed decisions about their development.

Methods

The authors propose a three-level Evaluation Indicators System for OSE sustainability, inspired by natural ecosystem assessment models.

  • Target Level: Defines the overall goal – measuring OSE sustainability.
  • Guideline Level: Identifies key indicators for sustainability: openness, stability, activity, and extensibility. These are derived from analyzing natural ecosystem sustainability factors and adapted for OSEs.
  • Evaluation Level: Breaks down the guideline indicators into measurable metrics based on participant activities within the ecosystem.

To quantify sustainability, they developed a Weight Calculation Method based on Information Contribution Value. This method assigns weights to indicators based on their dispersion and impact on the ecosystem's dynamic development. Subsequently, a Sustainability Assessment Model is proposed, which combines these weighted indicators to generate a composite sustainability index. This model uses a weighted sum of the guideline level indicators, where each guideline indicator is a weighted sum of its evaluation level indicators.

For predicting future sustainability, they employ the Gray Prediction Model, a technique suitable for time-series data with limited information, to forecast the composite sustainability index.

Datasets

The primary dataset used for empirical analysis is from the Stack Overflow (SO) ecosystem. The authors extracted data from 2010 to 2016, covering approximately 12.5 million questions, user activity logs, and tag information. This dataset is publicly available through Stack Overflow's data dump and the Stack Exchange Data Explorer.

Key Insights and Ideas

  • OSEs are like Natural Ecosystems: The paper draws a strong analogy between OSEs and natural ecosystems, suggesting that principles of ecological sustainability can be adapted to understand and measure OSE health.
  • Sustainability is Multi-faceted: Openness, stability, activity, and extensibility are identified as crucial, distinct dimensions of OSE sustainability, each requiring specific metrics.
  • Individual Indicator Trends Don't Tell the Whole Story: The analysis of SO revealed that while individual indicators might show different trends (e.g., openness increasing, activity decreasing), the overall sustainability requires a holistic view of all indicators.
  • Participant Activity is Key: The evaluation level indicators are directly tied to user actions and interactions, highlighting the importance of community engagement for ecosystem health.
  • Predictive Power: The application of the Gray Prediction Model demonstrates the potential to forecast OSE sustainability, offering valuable foresight for stakeholders.

Key Results

  • The proposed Evaluation Indicators System and Sustainability Assessment Model were applied to the Stack Overflow (SO) ecosystem.
  • Openness in SO showed a consistent upward trend from 2010-2016, indicating a healthy ability to attract new users and facilitate communication.
  • Stability remained relatively stable, with minor fluctuations, suggesting a consistent core of long-term contributors.
  • Activity showed a general downward trend, suggesting a potential need to strengthen user interaction and information exchange.
  • Extensibility demonstrated a significant increase, indicating SO's strong capacity for development and adaptation to evolving trends.
  • The Comprehensive Assessment Index for SO's sustainability showed a steady upward trend from 2010 to 2016, with a predicted continued increase from 2017-2021.
  • The Gray Prediction Model achieved high accuracy, with an average relative error of 4.39%, validating its predictive capability for OSE sustainability.

Limitations and Future Work

  • Scope of Empirical Study: The current study focuses solely on the Stack Overflow ecosystem. The authors acknowledge that the Evaluation Indicators System needs to be validated on a wider range of OSEs, particularly those with strong group collaboration mechanisms (like GitHub, SourceForge, Quora, Topcoder).
  • Indicator Adaptability: The authors note that the proposed evaluation indicators might need to be adjusted (added, removed, or replaced) based on the specific characteristics of different OSEs. For instance, "no bug fix time" was removed for SO.
  • Tool Development: The ultimate goal is to develop a practical sustainability measurement tool that facilitates user experience and enables comprehensive measurement of all indicators.
  • Ecosystem Recommendation: The framework could be extended for use in ecosystem recommendation systems and for predicting ecosystem sustainability in broader research fields.

Summary updated

BibTeX
@article{liao2018empirical,
  author = {Zhifang Liao and Libing Deng and Xiaoping Fan and Yan Zhang and Hui Liu and Xiaofei Qi and Yun Zhou},
  title = {{Empirical Research on the Evaluation Model and Method of Sustainability of the Open Source Ecosystem}},
  journal = {Symmetry},
  year = {2018},
  doi = {10.3390/sym10120747},
}
Figure