TL;DR
PTMTorrent is the first large-scale, multi-hub dataset designed to enable empirical research on pre-trained model (PTM) packages. It provides a standardized snapshot of nearly 16,000 models across five major registries (totaling ~61TB), allowing researchers to study the "model supply chain" much like they study traditional software ecosystems like NPM or PyPI.
What Problem Does This Paper Address?
In modern AI development, engineers rarely build deep neural networks from scratch. Instead, they download pre-trained models (PTMs) from "model hubs" and fine-tune them. While this accelerates development, it introduces new challenges in software engineering: How do we track model versions? How do we manage dependencies? Are there security risks in the model supply chain?
Mining this data is currently a nightmare for researchers because:
- Fragmentation: Models are scattered across many competing registries (Hugging Face, PyTorch Hub, etc.).
- Scale: Model weights are massive (often exceeding 1TB for a single package), making bulk downloads prohibitive.
- Access Barriers: Registries often impose rate limits or throttling that prevent large-scale data collection.
PTMTorrent solves this by providing a curated, standardized, and hosted dataset that researchers can access without hitting registry-side bottlenecks.
Methods
The authors built a pipeline to crawl and archive PTM packages from five major registries. Key technical steps included:
- Git Integration: For registries that support it, they performed full
git cloneoperations to capture the entire version history of the model packages. - Metadata Standardization: Since every hub uses a different format, the team developed a uniform data schema (using JSON Schema) to map disparate metadata—like model architectures, licenses, and training datasets—into a common format for cross-hub analysis.
- Infrastructure: To handle the massive 61TB footprint, they utilized Purdue University’s "Fortress" (a tape-based hierarchical storage system) and distributed the data via Globus, a high-performance data transfer service.
Datasets
The paper introduces the PTMTorrent (v1) dataset, which aggregates data from:
- Hugging Face: The top 10% most-downloaded models (12,401 models).
- Model Zoo: 3,245 models.
- ONNX Model Zoo: 185 models.
- PyTorch Hub: 49 models.
- Modelhub: 33 models.
The dataset includes not just the weights, but also model cards, documentation, and the underlying GitHub repository histories where available.
Key Insights and Ideas
- PTMs as Software Artifacts: The core philosophy is that a PTM is more than just a file of weights; it is a software package with a lifecycle, contributors, and dependencies.
- The "Top 10%" Strategy: Because Hugging Face is so vast (hundreds of thousands of models), the authors focused on the "top 10%" by downloads. This captures the models that actually impact the industry while keeping the storage requirements (relatively) sane.
- Cross-Hub Comparability: By creating a "parent" schema and "child" schemas for specific registries, the authors enable researchers to ask questions like, "How does documentation quality differ between PyTorch Hub and ONNX?"
Key Results
- Scale: The dataset contains 15,913 PTM packages and occupies approximately 61 Terabytes.
- Accessibility: By hosting the data on a high-performance storage system (HPSS), the authors provide a way for the scientific community to perform replicable research without being blocked by commercial registry rate limits.
- Tooling: The authors open-sourced their dataset generation tools, allowing others to update the dataset or add new hubs in the future.
Limitations and Future Work
- Bias Toward Popularity: The Hugging Face portion is biased toward the most-downloaded models, potentially missing "long-tail" or niche models.
- Missing Registries: Some hubs, like TensorFlow Hub, were excluded because they lacked robust version control features at the time of collection.
- Metadata Granularity: Some hubs (like ONNX) provide metadata in unstructured Markdown, making it harder to extract clean, tabular data compared to more structured registries.
- Future Directions: The authors suggest using PTMTorrent to study the PTM supply chain (e.g., identifying vulnerable dependencies), model evolution over time, and the development of better model recommendation systems.
Summary updated