Abstract
Due to the cost of developing and training deep learning models from scratch, machine learning engineers have begun to reuse pre-trained models (PTMs) and fine-tune them for downstream tasks. PTM registries known as "model hubs" support engineers in distributing and reusing deep learning models. PTM packages include pre-trained weights, documentation, model architectures, datasets, and metadata. Mining the information in PTM packages will enable the discovery of engineering phenomena and tools to support software engineers. However, accessing this information is difficult - there are many PTM registries, and both the registries and the individual packages may have rate limiting for accessing the data. We present an open-source dataset, PTMTorrent, to facilitate the evaluation and understanding of PTM packages. This paper describes the creation, structure, usage, and limitations of the dataset. The dataset includes a snapshot of 5 model hubs and a total of 15,913 PTM packages. These packages are represented in a uniform data schema for cross-hub mining. The PTMTorrent dataset (v1) is available at: https://app.globus.org/file-manager?origin_id=55e17a6e-9d8f-11ed-a2a2-8383522b48d9&origin_path=%2F~%2F. Our dataset generation tools are available on GitHub: https://doi.org/10.5281/zenodo.7570357.
Introduction
In modern AI development, engineers rarely build deep neural networks from scratch. Instead, they download pre-trained models (PTMs) from "model hubs" and fine-tune them. While this accelerates development, it introduces new challenges in software engineering: tracking model versions, managing dependencies, and assessing security risks in the model supply chain.
Mining this data is currently difficult for researchers due to fragmentation across competing registries (e.g., Hugging Face, PyTorch Hub), the massive scale of model weights (often exceeding 1TB per package), and access barriers like rate limits. PTMTorrent addresses these issues by providing a curated, standardized, and hosted dataset for large-scale empirical research.
Methodology and Dataset Creation
The authors developed a pipeline to crawl and archive PTM packages from five major registries. The process involves capturing the entire version history via Git integration where possible and mapping disparate metadata into a uniform data schema.
To handle the massive 61TB footprint, the dataset is hosted on Purdue University’s "Fortress" storage system and distributed via Globus, a high-performance data transfer service. This infrastructure allows researchers to perform replicable research without being blocked by commercial registry rate limits.
Dataset Characteristics
PTMTorrent (v1) aggregates data from five diverse hubs, focusing on the most impactful models. For Hugging Face, the authors captured the top 10% most-downloaded models to ensure the dataset remains manageable while covering the most relevant industry artifacts.
The dataset contains 15,913 PTM packages and occupies approximately 61 Terabytes. It includes not just weights, but also model cards, documentation, and underlying GitHub repository histories.
Research Opportunities and Insights
The core philosophy of PTMTorrent is treating PTMs as software artifacts with lifecycles, contributors, and dependencies. By standardizing metadata, researchers can conduct cross-hub comparisons, such as analyzing documentation quality differences between PyTorch Hub and ONNX.
Potential research directions include:
- PTM Supply Chain: Identifying vulnerable dependencies and tracking model evolution.
- Recommendation Systems: Developing better tools for model discovery.
- Ecosystem Analysis: Studying the "model supply chain" similarly to traditional software ecosystems like NPM or PyPI.
Limitations
The dataset has a bias toward popularity, particularly in the Hugging Face subset. Some registries, like TensorFlow Hub, were excluded due to a lack of robust version control features at the time of collection. Additionally, extracting structured data from hubs that use unstructured Markdown (like ONNX) remains a challenge.
Summary updated