Abstract
As deep learning models become increasingly complex and resource-intensive to train, the "Pre-Trained Model" (PTM) has emerged as the fundamental building block of modern AI software. Developers frequently leverage "model zoos" like Hugging Face to integrate sophisticated capabilities into their applications. However, the dynamics of this PTM supply chain—including documentation standards, licensing compatibility, and reuse patterns—remain poorly understood. This paper introduces PeaTMOSS, a massive, first-of-its-kind dataset designed to facilitate empirical research into the AI software ecosystem. PeaTMOSS links over 280,000 pre-trained models to more than 28,000 GitHub repositories that utilize them, providing a holistic view of model reuse and its associated risks.
Introduction
While traditional software dependency ecosystems (e.g., npm, PyPI) are well-mapped, the PTM supply chain has largely remained a "black box." Existing research often focuses solely on model registries, leaving a gap in our understanding of how these models are actually deployed in real-world open-source projects. Furthermore, model documentation is notoriously inconsistent; critical data regarding training sets, evaluation metrics, and carbon footprints are often trapped in unstructured "model cards" (README files). PeaTMOSS addresses these challenges by providing a structured, queryable database of PTM metadata and their downstream applications.
Methodology
The construction of PeaTMOSS involved a sophisticated multi-step pipeline designed to bridge the gap between model registries and code repositories:
- Registry Mining: The researchers scraped metadata for 281,638 PTMs from Hugging Face and PyTorch Hub.
- Downstream Linking: To identify real-world usage, the team defined "signatures" of PTM integration (e.g., specific Python library imports and calls like
AutoModel.from_pretrained()). Using Sourcegraph, they searched GitHub for these signatures, identifying 28,575 relevant repositories. - LLM-Powered Extraction: To structure the messy data found in model cards, the authors employed GPT-3.5 and GPT-4. They developed an "Accurate" pipeline that transforms free-form text into structured JSON metadata, capturing details such as parameter counts and hardware requirements.
- Static Analysis: The Scalpel framework was used to perform static analysis on the identified GitHub repositories, confirming that the models were actively invoked within the source code.
The PeaTMOSS Dataset
The primary contribution of this work is the PeaTMOSS dataset, which is offered in two distinct versions to suit different research needs:
- Metadata Version (7.12 GB): A SQLite database containing structured metadata for 281,638 PTMs, 28,575 GitHub projects, and 44,337 verified links between them.
- Full Version (48.2 TB): A comprehensive archive including snapshots of popular models (those with >50 monthly downloads) and the complete git history of the downstream GitHub projects.
Key Insights and Empirical Analysis
Initial analysis of the PeaTMOSS dataset reveals several critical trends in the AI supply chain:
The Documentation Gap
Despite the use of advanced LLM extraction, a significant lack of transparency persists. Fewer than 10% of model cards mention carbon emissions or grant sponsorship, and less than half provide links to original research papers or source repositories. This lack of documentation poses a challenge for reproducibility and ethical auditing.
Explosive Growth and the "Llama Effect"
PTM usage is doubling year-over-year. Between August 2022 and August 2023, the number of PTM packages on Hugging Face grew from 124,000 to 282,000. The researchers also noted a massive spike in median parameter counts for NLP models starting in early 2023, a trend largely attributed to the release of landmark models like Llama 2.
Licensing and Safety Risks
By cross-referencing model and application licenses, the authors identified that 0.24% of PTM-application pairs exhibit license incompatibilities (e.g., a copyleft model used in a way that violates its terms). While the percentage is small, the scale of the ecosystem suggests thousands of potential legal and compliance issues.
Conclusion and Future Work
PeaTMOSS provides the foundation for a new field of "AI Supply Chain" research. While current mapping relies on static analysis—which may miss models loaded dynamically—it offers the most comprehensive view of PTM reuse to date. Future work will focus on identifying "code smells" specific to PTM-enabled applications and investigating the "phylogeny" of models as they are fine-tuned and evolved across the open-source landscape.
Summary updated