PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software

Wenxin Jiang, Jerin Yasmin, Jason Jones, Nicholas Synovic, Jiashen Kuo, Nathaniel Bielanski, Yuan Tian, George K. Thiruvathukal, James C. Davis

Abstract

As deep learning models become increasingly complex and resource-intensive to train, the "Pre-Trained Model" (PTM) has emerged as the fundamental building block of modern AI software. Developers frequently leverage "model zoos" like Hugging Face to integrate sophisticated capabilities into their applications. However, the dynamics of this PTM supply chain—including documentation standards, licensing compatibility, and reuse patterns—remain poorly understood. This paper introduces PeaTMOSS, a massive, first-of-its-kind dataset designed to facilitate empirical research into the AI software ecosystem. PeaTMOSS links over 280,000 pre-trained models to more than 28,000 GitHub repositories that utilize them, providing a holistic view of model reuse and its associated risks.

Introduction

While traditional software dependency ecosystems (e.g., npm, PyPI) are well-mapped, the PTM supply chain has largely remained a "black box." Existing research often focuses solely on model registries, leaving a gap in our understanding of how these models are actually deployed in real-world open-source projects. Furthermore, model documentation is notoriously inconsistent; critical data regarding training sets, evaluation metrics, and carbon footprints are often trapped in unstructured "model cards" (README files). PeaTMOSS addresses these challenges by providing a structured, queryable database of PTM metadata and their downstream applications.

Methodology

The construction of PeaTMOSS involved a sophisticated multi-step pipeline designed to bridge the gap between model registries and code repositories:

  1. Registry Mining: The researchers scraped metadata for 281,638 PTMs from Hugging Face and PyTorch Hub.
  2. Downstream Linking: To identify real-world usage, the team defined "signatures" of PTM integration (e.g., specific Python library imports and calls like AutoModel.from_pretrained()). Using Sourcegraph, they searched GitHub for these signatures, identifying 28,575 relevant repositories.
  3. LLM-Powered Extraction: To structure the messy data found in model cards, the authors employed GPT-3.5 and GPT-4. They developed an "Accurate" pipeline that transforms free-form text into structured JSON metadata, capturing details such as parameter counts and hardware requirements.
  4. Static Analysis: The Scalpel framework was used to perform static analysis on the identified GitHub repositories, confirming that the models were actively invoked within the source code.
The PTM supply chain: from engineers to downstream applications.

The PeaTMOSS Dataset

The primary contribution of this work is the PeaTMOSS dataset, which is offered in two distinct versions to suit different research needs:

  • Metadata Version (7.12 GB): A SQLite database containing structured metadata for 281,638 PTMs, 28,575 GitHub projects, and 44,337 verified links between them.
  • Full Version (48.2 TB): A comprehensive archive including snapshots of popular models (those with >50 monthly downloads) and the complete git history of the downstream GitHub projects.
The PeaTMOSS database schema, illustrating the linkage between PTM metadata and GitHub project data.

Key Insights and Empirical Analysis

Initial analysis of the PeaTMOSS dataset reveals several critical trends in the AI supply chain:

The Documentation Gap

Despite the use of advanced LLM extraction, a significant lack of transparency persists. Fewer than 10% of model cards mention carbon emissions or grant sponsorship, and less than half provide links to original research papers or source repositories. This lack of documentation poses a challenge for reproducibility and ethical auditing.

Explosive Growth and the "Llama Effect"

PTM usage is doubling year-over-year. Between August 2022 and August 2023, the number of PTM packages on Hugging Face grew from 124,000 to 282,000. The researchers also noted a massive spike in median parameter counts for NLP models starting in early 2023, a trend largely attributed to the release of landmark models like Llama 2.

Licensing and Safety Risks

By cross-referencing model and application licenses, the authors identified that 0.24% of PTM-application pairs exhibit license incompatibilities (e.g., a copyleft model used in a way that violates its terms). While the percentage is small, the scale of the ecosystem suggests thousands of potential legal and compliance issues.

Conclusion and Future Work

PeaTMOSS provides the foundation for a new field of "AI Supply Chain" research. While current mapping relies on static analysis—which may miss models loaded dynamically—it offers the most comprehensive view of PTM reuse to date. Future work will focus on identifying "code smells" specific to PTM-enabled applications and investigating the "phylogeny" of models as they are fine-tuned and evolved across the open-source landscape.

Summary updated

Abstract

The development and training of deep learning models have become increasingly costly and complex. Consequently, software engineers are adopting pre-trained models (PTMs) for their downstream applications. The dynamics of the PTM supply chain remain largely unexplored, signaling a clear need for structured datasets that document not only the metadata but also the subsequent applications of these models. Without such data, the MSR community cannot comprehensively understand the impact of PTM adoption and reuse. This paper presents the PeaTMOSS dataset, which comprises metadata for 281,638 PTMs and detailed snapshots for all PTMs with over 50 monthly downloads (14,296 PTMs), along with 28,575 open-source software repositories from GitHub that utilize these models. Additionally, the dataset includes 44,337 mappings from 15,129 downstream GitHub repositories to the 2,530 PTMs they use. To enhance the dataset's comprehensiveness, we developed prompts for a large language model to automatically extract model metadata, including the model's training datasets, parameters, and evaluation metrics. Our analysis of this dataset provides the first summary statistics for the PTM supply chain, showing the trend of PTM development and common shortcomings of PTM package documentation. Our example application reveals inconsistencies in software licenses across PTMs and their dependent projects. PeaTMOSS lays the foundation for future research, offering rich opportunities to investigate the PTM supply chain. We outline mining opportunities on PTMs, their downstream usage, and cross-cutting questions.
BibTeX
@inproceedings{jiang2024peatmoss,
  author = {Wenxin Jiang and Jerin Yasmin and Jason Jones and Nicholas Synovic and Jiashen Kuo and Nathaniel Bielanski and Yuan Tian and George K. Thiruvathukal and James C. Davis},
  title = {{PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software}},
  booktitle = {Msr},
  year = {2024},
  doi = {10.1145/3643991.3644907},
}

Loading related content...

Figure