"I see models being a whole other thing": An Empirical Study of Pre-Trained Model Naming Conventions and A Tool for Enhancing Naming Consistency

Wenxin Jiang, Mingyu Kim, Chingwo Cheung, Heesoo Kim, George K. Thiruvathukal, James C. Davis

TL;DR

This paper presents the first systematic study of how we name Pre-Trained Models (PTMs) and introduces DARA, an automated tool that detects naming inconsistencies by analyzing a model's internal architecture. The researchers found a significant mismatch between what engineers want in a model name and current industry practices, but proved that architectural data alone can identify mislabeled models with up to 99% accuracy.

What Problem Does This Paper Address?

In traditional software engineering, a package name like requests or chalk tells you roughly what a tool does or provides a catchy brand. However, in the world of Deep Learning, a model's name is often its primary "interface." Engineers rely on names to infer critical details: Is this a BERT model? Was it trained on the SQuAD dataset? Is it the "large" or "base" version?

Unfortunately, PTM naming is currently a "Wild West." Inconsistent metadata and misleading names make it difficult for researchers to discover the right models and, more importantly, create security risks like "typosquatting" or the accidental reuse of models with hidden backdoors. As shown in Figure 1, even major players like Meta can have mismatches between a model's identifier and its actual underlying architecture, leading to confusion and wasted engineering hours.

Fig. 1: Example of a PTM metadata inconsistency for a PTM on Hugging Face. The top part shows the PTM identifier, and the bottom part displays the PTM metadata (i.e., architecture and model type) and the package content. On the right (blue boxes) is a four-step reuse process adapted from (Jiang et al. 2023b). In this case, Facebook (Meta) named the model nllb due to the introduction of a new tokenizer, while other models on Hugging Face typically name models based on architecture changes rather than changes in tokenizers. This mismatch between the identifier and the architecture metadata impacts the new model’s searchability, perceived reliability, and ultimately reusability in the PTM reuse process.

Methods

The authors employed a three-pronged, mixed-methods approach:

  1. User Survey: They surveyed 108 Hugging Face users to understand their preferences and the challenges they face with current naming conventions.
  2. Large-Scale Repository Mining: Using GPT-4, they analyzed over 14,000 PTM identifiers from the Hugging Face registry to extract common "naming elements" (like architecture, size, and task) and identify prevalent naming conventions.
  3. DARA (DNN ARchitecture Assessment): They developed a novel tool to automatically verify if a model's name matches its "ground truth"—its weights and layers. DARA converts a model into an Abstract PTM Architecture (APTM), extracts features (using n-grams or Transformers), and uses a classifier to predict the correct model_type, architecture, and task.

Datasets

  • PeaTMOSS: A large-scale dataset of PTMs used for the repository mining study (14,296 models).
  • Hugging Face Registry: The primary source for the survey participants and the models evaluated by DARA.
  • DARA Evaluation Set: A curated subset of 4,108 PTM packages across 184 distinct architectures and 81 model types.

Key Insights and Ideas

  • PTMs are "Hybrid" Artifacts: The study reveals that PTMs aren't just software; they are treated more like hardware components. Their names function like a "spec sheet."
  • The Semantic Gap: There is a clear misalignment between producers and consumers. While engineers strongly prefer names that combine implementation (how it works) and application (what it does), nearly 60% of models in the wild only list implementation details.
  • Architecture as Truth: The most profound insight is the "Engineering Hypothesis": a model's internal wiring (layer connections and parameters) is a reliable enough fingerprint to detect naming fraud or errors without needing to see the training data or documentation.

This difference in naming complexity between PTMs and traditional software is starkly visible when comparing top packages across registries, as seen in the table below:

Table 1: Top 10 package names by weekly downloads from NPM, PyPI, and Hug- ging Face. Compared to the NPM and PyPI packages (traditional software), the names of Hugging Face packages (PTMs) appear to convey more information about the package content. This paper is the first to study this phenomenon. Popularity data is as of September 2024.

Key Results

  • High Detection Accuracy: DARA achieved a staggering 99% accuracy in identifying the correct model_type based solely on architectural features.
  • Task and Architecture Identification: The tool performed moderately well for more granular details, reaching 72% accuracy for specific architectures and 76% for intended tasks.
  • Efficiency vs. Accuracy: The researchers found a clear trade-off. Simple N-gram models are incredibly fast (processing 28,000+ models per second), making them ideal for real-time "linting" during model uploads, while Transformer-based versions are more accurate but slower.

Limitations and Future Work

While DARA is a powerful proof-of-concept, it has its limits. It struggles to distinguish between models that share the exact same architecture but differ in training regimes or datasets (e.g., two identical ResNet models trained on different data).

The authors suggest that future versions of DARA could incorporate "multimodal" features—combining architectural analysis with natural language processing of model cards—to close this gap. They also envision DARA-like tools becoming a standard part of the AI supply chain, acting as an automated "security guard" to prevent model plagiarism and supply chain attacks.

Summary updated

Abstract

As innovation in deep learning continues, many engineers are incorporating Pre-Trained Models (PTMs) as components in computer systems. Some PTMs are foundation models, and others are fine-tuned variations adapted to different needs. When these PTMs are named well, it facilitates model discovery and reuse. However, prior research has shown that model names are not always well chosen and can sometimes be inaccurate and misleading. The naming practices for PTM packages have not been systematically studied, which hampers engineers' ability to efficiently search for and reliably reuse these models. In this paper, we conduct the first empirical investigation of PTM naming practices in the Hugging Face PTM registry. We begin by reporting on a survey of 108 Hugging Face users, highlighting differences from traditional software package naming and presenting findings on PTM naming practices. The survey results indicate a mismatch between engineers' preferences and current practices in PTM naming. We then introduce DARA, the first automated DNN ARchitecture Assessment technique designed to detect PTM naming inconsistencies. Our results demonstrate that architectural information alone is sufficient to detect these inconsistencies, achieving an accuracy of 94% in identifying model types and promising performance (over 70%) in other architectural metadata as well. We also highlight potential use cases for automated naming tools, such as model validation, PTM metadata generation and verification, and plagiarism detection. Our study provides a foundation for automating naming inconsistency detection. Finally, we envision future work focusing on automated tools for standardizing package naming, improving model selection and reuse, and strengthening the security of the PTM supply chain.
BibTeX
@article{jiang2025i,
  author = {Wenxin Jiang and Mingyu Kim and Chingwo Cheung and Heesoo Kim and George K. Thiruvathukal and James C. Davis},
  title = {{"I see models being a whole other thing": An Empirical Study of Pre-Trained Model Naming Conventions and A Tool for Enhancing Naming Consistency}},
  journal = {Empir. Softw. eng},
  year = {2025},
  doi = {10.1007/s10664-025-10711-4},
}
Figure