TL;DR
This paper presents the first systematic study of how we name Pre-Trained Models (PTMs) and introduces DARA, an automated tool that detects naming inconsistencies by analyzing a model's internal architecture. The researchers found a significant mismatch between what engineers want in a model name and current industry practices, but proved that architectural data alone can identify mislabeled models with up to 99% accuracy.
What Problem Does This Paper Address?
In traditional software engineering, a package name like requests or chalk tells you roughly what a tool does or provides a catchy brand. However, in the world of Deep Learning, a model's name is often its primary "interface." Engineers rely on names to infer critical details: Is this a BERT model? Was it trained on the SQuAD dataset? Is it the "large" or "base" version?
Unfortunately, PTM naming is currently a "Wild West." Inconsistent metadata and misleading names make it difficult for researchers to discover the right models and, more importantly, create security risks like "typosquatting" or the accidental reuse of models with hidden backdoors. As shown in Figure 1, even major players like Meta can have mismatches between a model's identifier and its actual underlying architecture, leading to confusion and wasted engineering hours.
Methods
The authors employed a three-pronged, mixed-methods approach:
- User Survey: They surveyed 108 Hugging Face users to understand their preferences and the challenges they face with current naming conventions.
- Large-Scale Repository Mining: Using GPT-4, they analyzed over 14,000 PTM identifiers from the Hugging Face registry to extract common "naming elements" (like architecture, size, and task) and identify prevalent naming conventions.
- DARA (DNN ARchitecture Assessment): They developed a novel tool to automatically verify if a model's name matches its "ground truth"—its weights and layers. DARA converts a model into an Abstract PTM Architecture (APTM), extracts features (using n-grams or Transformers), and uses a classifier to predict the correct
model_type,architecture, andtask.
Datasets
- PeaTMOSS: A large-scale dataset of PTMs used for the repository mining study (14,296 models).
- Hugging Face Registry: The primary source for the survey participants and the models evaluated by DARA.
- DARA Evaluation Set: A curated subset of 4,108 PTM packages across 184 distinct architectures and 81 model types.
Key Insights and Ideas
- PTMs are "Hybrid" Artifacts: The study reveals that PTMs aren't just software; they are treated more like hardware components. Their names function like a "spec sheet."
- The Semantic Gap: There is a clear misalignment between producers and consumers. While engineers strongly prefer names that combine implementation (how it works) and application (what it does), nearly 60% of models in the wild only list implementation details.
- Architecture as Truth: The most profound insight is the "Engineering Hypothesis": a model's internal wiring (layer connections and parameters) is a reliable enough fingerprint to detect naming fraud or errors without needing to see the training data or documentation.
This difference in naming complexity between PTMs and traditional software is starkly visible when comparing top packages across registries, as seen in the table below:
Key Results
- High Detection Accuracy: DARA achieved a staggering 99% accuracy in identifying the correct
model_typebased solely on architectural features. - Task and Architecture Identification: The tool performed moderately well for more granular details, reaching 72% accuracy for specific architectures and 76% for intended tasks.
- Efficiency vs. Accuracy: The researchers found a clear trade-off. Simple N-gram models are incredibly fast (processing 28,000+ models per second), making them ideal for real-time "linting" during model uploads, while Transformer-based versions are more accurate but slower.
Limitations and Future Work
While DARA is a powerful proof-of-concept, it has its limits. It struggles to distinguish between models that share the exact same architecture but differ in training regimes or datasets (e.g., two identical ResNet models trained on different data).
The authors suggest that future versions of DARA could incorporate "multimodal" features—combining architectural analysis with natural language processing of model cards—to close this gap. They also envision DARA-like tools becoming a standard part of the AI supply chain, acting as an automated "security guard" to prevent model plagiarism and supply chain attacks.
Summary updated