Cross-ecosystem categorization: A manual-curation protocol for the categorization of Java Maven libraries along Python PyPI Topics

Ranindya Paramitha, Yuan Feng, Fabio Massacci, Carlos E. Budde

TL;DR

Researchers have developed a human-guided protocol to categorize software libraries across different ecosystems (like Java and Python) using a single, unified taxonomy. By mapping 256 vulnerable Java libraries to Python’s "PyPI Topics," they’ve created a "Rosetta Stone" that allows for unbiased cross-language security and popularity comparisons.

What Problem Does This Paper Address?

In the world of Software Engineering (SE), we often want to compare different ecosystems. For example: "Is Java more prone to security vulnerabilities than Python?"

However, making a fair comparison is surprisingly difficult. Each ecosystem uses its own "organic" categorization system. Maven (Java) might have over 150 groups with names like "Atlassian Plugins," while PyPI (Python) uses a set of 24 functional "Topics." If you try to compare the most popular libraries in both, you might find that Python's top libraries are for "Software Development" while Java's are for "Internet" services. Because different types of software have different risk profiles, comparing them directly leads to skewed results—essentially comparing apples to oranges.

The authors address this "functional fingerprint" gap by creating a systematic way to label any software library, regardless of its native ecosystem, using a common set of functional categories.

Methods

The authors designed a four-stage manual curation protocol:

  1. Step 0: Libraries Definition: Selecting the target libraries based on clear criteria (e.g., having a high-severity vulnerability).
  2. Step 1: Source Collection: Using "data triangulation" to gather information. For each library, they look at four sources: the Maven Central entry, the source code repository (GitHub/GitLab), the official website, and any available documentation or wikis.
  3. Step 2: Assessment (Categorization): At least two human "assessors" independently assign up to two categories from the 24 PyPI Topics to each library.
  4. Step 3: Arbitration: An "arbitrator" resolves any conflicts. If the assessors disagree, the arbitrator reviews the sources and makes a final call.

A key innovation here is the Class Revision phase. The authors grouped the 24 fine-grained topics into two coarse classes: Remote Network (high internet exposure) and Local (low exposure). They then used security metadata (like the CVSS Attack Vector) to flag "suspicious" categorizations for a second look, ensuring the functional label matched the library's actual security context.

Datasets

The study focused on a newly curated dataset of 256 Java/Maven libraries.

  • Selection Criteria: Libraries had to be Open Source (FOSS) and have at least one "High" or "Critical" vulnerability (CVSS score $\ge$ 7) listed in the Snyk vulnerability database.
  • Comparison Set: They compared this Java dataset against the Top-256 popular Python/PyPI libraries to demonstrate the inherent bias in cross-ecosystem data.

The authors have made the full protocol, the 256-library dataset, and the scripts used to generate the statistics available as open data on Zenodo.

Key Insights and Ideas

  • The Popularity Bias: The paper highlights a major pitfall in empirical studies: popular libraries in one language often serve completely different functions than popular libraries in another. Without a common taxonomy, researchers might mistakenly attribute differences to the language itself rather than the type of software being written.
  • Humans are Still Better at Inference: While machine learning (ML) and natural language processing (NLP) are getting better, the authors argue that humans are still superior at inferring a library's "abstract function" from messy, varied documentation.
  • The "Internet-Facing" Factor: A staggering 53% of the vulnerable Java libraries were categorized as "Internet." This confirms that libraries exposed to the web are under higher scrutiny and more likely to have reported severe vulnerabilities.
  • Standardizing the "Why": By requiring assessors to use a specific set of rules and sources, the protocol moves away from "ad-hoc" researcher decisions toward a reproducible scientific process.

Key Results

  • Agreement Levels: The two human assessors agreed on at least one category for 57.8% of the libraries. Using Fleiss’ kappa ($\kappa = 0.382$), they achieved "fair agreement," which is respectable given the complexity of the task.
  • Functional Distribution: The Java dataset was heavily skewed toward Internet (135 libraries) and Database (30 libraries). In contrast, popular Python libraries are skewed toward Software Development.
  • Protocol Efficiency: With a minimal team of three (two assessors, one arbitrator), the pipelined approach allowed for the systematic categorization of hundreds of libraries in a reasonable timeframe.

Limitations and Future Work

  • Labor Intensive: The primary drawback is that this is a manual process. It is not intended for million-library datasets but rather for creating high-quality "ground truth" samples.
  • Subjectivity: Despite the guidelines, human interpretation remains subjective. The authors suggest that adding more assessors could further reduce bias.
  • Scaling Up: The authors envision this dataset being used to train ML and NLP models. Once an AI is trained on this human-curated "ground truth," it could potentially scale the categorization to the millions of libraries found in Maven and PyPI.
  • Ecosystem Expansion: Future work involves applying this protocol to other ecosystems like npm (JavaScript) or RubyGems to see if the same security-functional patterns hold true across the entire FOSS landscape.

Summary updated

Abstract

Context: Software of different functional categories, such as text processing vs. networking, has different profiles in terms of metrics like security and updates. Using popularity to compare e.g. Java vs. Python libraries might give a skewed perspective, as the categories of the most popular software vary from one ecosystem to the next. How can one compare libraries datasets across software ecosystems, when not even the category names are uniform among them? Objective: We study how to generate a language-agnostic categorisation of software by functional purpose, that enables cross-ecosystem studies of libraries datasets. This provides the functional fingerprint information needed for software metrics comparisons. Method: We designed and implemented a human-guided protocol to categorise libraries from software ecosystems. Category names mirror PyPI Topic classifiers, but the protocol is generic and can be applied to any ecosystem. We demonstrate it by categorising 256 Java/Maven libraries with severe security vulnerabilities. Results: The protocol allows three or more people to categorise any number of libraries. The categorisation produced is functional-oriented and language-agnostic. The Java/Maven dataset demonstration resulted in a majority of Internet-oriented libraries, coherent with its selection by severe vulnerabilities. To allow replication and updates, we make the dataset and the protocol individual steps available as open data. Conclusions: Libraries categorisation by functional purpose is feasible with our protocol, which produced the fingerprint of a 256-libraries Java dataset. While this was labour intensive, humans excel in the required inference tasks, so full automation of the process is not envisioned. However, results can provide the ground truth needed for machine learning in large-scale cross-ecosystem empirical studies.
BibTeX
@misc{paramitha2024cross,
  author = {Ranindya Paramitha and Yuan Feng and Fabio Massacci and Carlos E. Budde},
  title = {{Cross-ecosystem categorization: A manual-curation protocol for the categorization of Java Maven libraries along Python PyPI Topics}},
  year = {2024},
  eprint = {2403.06300},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.06300},
}
Figure