PTMTorrent: A Dataset for Mining Open-source Pre-trained Model Packages

Wenxin Jiang, Nicholas Synovic, Purvish Jajal, Taylor R. Schorlemmer, Arav Tewari, Bhavesh Pareek, George K. Thiruvathukal, James C. Davis

TL;DR

PTMTorrent is the first large-scale, multi-hub dataset designed to enable empirical research on pre-trained model (PTM) packages. It provides a standardized snapshot of nearly 16,000 models across five major registries (totaling ~61TB), allowing researchers to study the "model supply chain" much like they study traditional software ecosystems like NPM or PyPI.

What Problem Does This Paper Address?

In modern AI development, engineers rarely build deep neural networks from scratch. Instead, they download pre-trained models (PTMs) from "model hubs" and fine-tune them. While this accelerates development, it introduces new challenges in software engineering: How do we track model versions? How do we manage dependencies? Are there security risks in the model supply chain?

Mining this data is currently a nightmare for researchers because:

  1. Fragmentation: Models are scattered across many competing registries (Hugging Face, PyTorch Hub, etc.).
  2. Scale: Model weights are massive (often exceeding 1TB for a single package), making bulk downloads prohibitive.
  3. Access Barriers: Registries often impose rate limits or throttling that prevent large-scale data collection.

PTMTorrent solves this by providing a curated, standardized, and hosted dataset that researchers can access without hitting registry-side bottlenecks.

Methods

The authors built a pipeline to crawl and archive PTM packages from five major registries. Key technical steps included:

  • Git Integration: For registries that support it, they performed full git clone operations to capture the entire version history of the model packages.
  • Metadata Standardization: Since every hub uses a different format, the team developed a uniform data schema (using JSON Schema) to map disparate metadata—like model architectures, licenses, and training datasets—into a common format for cross-hub analysis.
  • Infrastructure: To handle the massive 61TB footprint, they utilized Purdue University’s "Fortress" (a tape-based hierarchical storage system) and distributed the data via Globus, a high-performance data transfer service.

Datasets

The paper introduces the PTMTorrent (v1) dataset, which aggregates data from:

  • Hugging Face: The top 10% most-downloaded models (12,401 models).
  • Model Zoo: 3,245 models.
  • ONNX Model Zoo: 185 models.
  • PyTorch Hub: 49 models.
  • Modelhub: 33 models.

The dataset includes not just the weights, but also model cards, documentation, and the underlying GitHub repository histories where available.

Key Insights and Ideas

  • PTMs as Software Artifacts: The core philosophy is that a PTM is more than just a file of weights; it is a software package with a lifecycle, contributors, and dependencies.
  • The "Top 10%" Strategy: Because Hugging Face is so vast (hundreds of thousands of models), the authors focused on the "top 10%" by downloads. This captures the models that actually impact the industry while keeping the storage requirements (relatively) sane.
  • Cross-Hub Comparability: By creating a "parent" schema and "child" schemas for specific registries, the authors enable researchers to ask questions like, "How does documentation quality differ between PyTorch Hub and ONNX?"

Key Results

  • Scale: The dataset contains 15,913 PTM packages and occupies approximately 61 Terabytes.
  • Accessibility: By hosting the data on a high-performance storage system (HPSS), the authors provide a way for the scientific community to perform replicable research without being blocked by commercial registry rate limits.
  • Tooling: The authors open-sourced their dataset generation tools, allowing others to update the dataset or add new hubs in the future.

Limitations and Future Work

  • Bias Toward Popularity: The Hugging Face portion is biased toward the most-downloaded models, potentially missing "long-tail" or niche models.
  • Missing Registries: Some hubs, like TensorFlow Hub, were excluded because they lacked robust version control features at the time of collection.
  • Metadata Granularity: Some hubs (like ONNX) provide metadata in unstructured Markdown, making it harder to extract clean, tabular data compared to more structured registries.
  • Future Directions: The authors suggest using PTMTorrent to study the PTM supply chain (e.g., identifying vulnerable dependencies), model evolution over time, and the development of better model recommendation systems.

Summary updated

Abstract

Due to the cost of developing and training deep learning models from scratch, machine learning engineers have begun to reuse pre-trained models (PTMs) and fine-tune them for downstream tasks. PTM registries known as “model hubs” support engineers in distributing and reusing deep learning models. PTM packages include pre-trained weights, documentation, model architectures, datasets, and metadata. Mining the information in PTM packages will enable the discovery of engineering phenomena and tools to support software engineers. However, accessing this information is difficult — there are many PTM registries, and both the registries and the individual packages may have rate limiting for accessing the data.We present an open-source dataset, PTMTorrent, to facilitate the evaluation and understanding of PTM packages. This paper describes the creation, structure, usage, and limitations of the dataset. The dataset includes a snapshot of 5 model hubs and a total of 15,913 PTM packages. These packages are represented in a uniform data schema for cross-hub mining. We describe prior uses of this data and suggest research opportunities for mining using our dataset.The PTMTorrent dataset (v1) is available at: https://app.globus.org/file-manager?origin_id=55e17a6e-9d8f-11ed-a2a2-8383522b48d9&origin_path=%2F%7E%2F.Our dataset generation tools are available on GitHub: https://doi.org/10.5281/zenodo.7570357
BibTeX
@inproceedings{jiang2023ptmtorrent,
  author = {Wenxin Jiang and Nicholas Synovic and Purvish Jajal and Taylor R. Schorlemmer and Arav Tewari and Bhavesh Pareek and George K. Thiruvathukal and James C. Davis},
  title = {{PTMTorrent: A Dataset for Mining Open-source Pre-trained Model Packages}},
  booktitle = {MSR},
  year = {2023},
  url = {https://openreview.net/forum?id=NBtwkbxYxRs},
}

Loading related content...

Figure