OSSMETER: Automated Measurement and Analysis of Open Source Software

Bruno Almeida, Sophia Ananiadou, Alessandra Bagnato, Alberto Berreteaga Barbero, Juri Di Rocco, Davide Di Ruscio, Dimitrios S. Kolovos, Ioannis Korkontzelos, Scott Hansen, Pedro Maló, Nikolaos Drivalos, Richard F. Paige, Jurgen J. Vinju

Introduction

Adopting open-source software (OSS) is a strategic decision for engineering teams, yet evaluating the health and long-term viability of a project remains a significant challenge. Decision-makers often struggle to determine if a project is well-maintained, if its community is responsive, or if critical bugs are addressed in a timely manner. OSSMETER is an automated platform designed to bridge this information gap by providing a comprehensive, multi-dimensional view of OSS project maturity and activity.

Problem Statement

Currently, assessing project health requires a manual and fragmented deep dive into various sources, including GitHub commits, Jira tickets, and mailing list archives. While existing automated tools can calculate source code metrics like cyclomatic complexity, they often neglect the "human" side of software development—community support and developer engagement—which are vital for a project's sustainability. OSSMETER addresses this by integrating data from source code repositories, bug trackers, and communication channels into a unified analysis framework.

System Architecture

The OSSMETER platform is built on a cloud-based, extensible OSGi (plug-in) architecture. Its design follows a multi-layered approach to handle the complexities of diverse data sources and analysis requirements.

The OSSMETER system architecture, showing the flow from raw data persistence to the REST API and Web Application.

The architecture consists of several key components:

  • Persistence Layer: Manages the storage of raw data and calculated metrics.
  • Fact and Metric Providers: "Fact Providers" extract raw data from various forges, while "Metric Providers" calculate specific indicators of project health.
  • Domain Modeling: Formal metamodels represent the OSS lifecycle, allowing the platform to act as a "universal translator" between different hosting environments (e.g., GitHub vs. Eclipse).
  • Analysis Engines: These include language-specific source code analyzers and Natural Language Processing (NLP) tools for text mining communication channels.

Evaluation and Datasets

To validate the platform's effectiveness and scalability, the authors evaluated OSSMETER against 11 diverse, real-world OSS projects. These projects were selected to represent a broad spectrum of sizes, ages, and hosting environments.

A list of the 11 projects used in the evaluation, including Odoo, Joomla, and Drupal, detailing their lines of code (LOC), age, and commit history.

The dataset included large-scale projects like Odoo (over 2 million lines of code) and Joomla, as well as specialized tools like Epsilon and ATL. This variety demonstrated the platform's ability to handle different metadata formats and scale to projects with nearly 100,000 commits.

Key Insights and Results

The evaluation revealed several critical insights into the measurement of open-source health:

  • Quantifying Community Health: By applying NLP and sentiment analysis to forums and mailing lists, OSSMETER demonstrates that "soft" metrics—such as developer responsiveness and community mood—can be quantified as rigorously as technical code metrics.
  • The "Factoid" Concept: To make complex data accessible to non-technical stakeholders, OSSMETER aggregates granular metrics into "Factoids"—intuitive four-star ratings that summarize project aspects like "community activity" or "code quality."
  • Engineering Depth: The OSSMETER platform itself represents a significant engineering effort, comprising over 650,000 lines of code and 1,800 commits as of mid-2015.
  • Public Utility: The platform provides a REST API, enabling researchers and developers to leverage OSSMETER’s quality indicators for their own tools and studies.

Conclusion

OSSMETER provides a robust, automated solution for evaluating the multi-faceted health of open-source projects. While the platform faces challenges related to metadata inconsistency across different hosting "forges," its extensible architecture and use of formal metamodels provide a strong foundation for future development. Beyond open source, the authors suggest that the platform's ability to track code evolution and team communication could be highly valuable for monitoring proprietary software projects within corporate environments, helping to identify at-risk internal projects before they fail.

Summary updated

Abstract

Deciding whether an open source software (OSS) meets the required standards for adoption in terms of quality, maturity, activity of development and user support is not a straightforward process. It involves analysing various sources of information, including the project’s source code repositories, communication channels, and bug tracking systems. OSSMETER extends state-of-the-art tech-niques in the field of automated analysis and measurement of open-source soft-ware (OSS), and develops a platform that supports decision makers in the pro-cess of discovering, comparing, assessing and monitoring the health, quality, im-pact and activity of opensource software. To achieve this, OSSMETER computes trustworthy quality indicators by performing advanced analysis and integration of information from diverse sources including the project metadata, source code
BibTeX
@inproceedings{almeida2015ossmeter,
  author = {Bruno Almeida and Sophia Ananiadou and Alessandra Bagnato and Alberto Berreteaga Barbero and Juri Di Rocco and Davide Di Ruscio and Dimitrios S. Kolovos and Ioannis Korkontzelos and Scott Hansen and Pedro Maló and Nikolaos Drivalos and Richard F. Paige and Jurgen J. Vinju},
  title = {{OSSMETER: Automated Measurement and Analysis of Open Source Software}},
  booktitle = {STAF Projects Showcase},
  year = {2015},
  url = {https://openreview.net/forum?id=T9ifTgzXfRi},
}
Figure