"I see models being a whole other thing": An Empirical Study of Pre-Trained Model Naming Conventions and A Tool for Enhancing Naming Consistency

Wenxin Jiang, Chingwo Cheung, George K. Thiruvathukal, James C. Davis, Thiruvathukal, George K., Davis, James C.

Abstract

As innovation in deep learning continues, many engineers are incorporating Pre-Trained Models (PTMs) as components in computer systems. Some PTMs are foundation models, and others are fine-tuned variations adapted to different needs. When these PTMs are named well, it facilitates model discovery and reuse. However, prior research has shown that model names are not always well chosen and can sometimes be inaccurate and misleading. The naming practices for PTM packages have not been systematically studied, which hampers engineers' ability to efficiently search for and reliably reuse these models. In this paper, we conduct the first empirical investigation of PTM naming practices in the Hugging Face PTM registry. We begin by reporting on a survey of 108 Hugging Face users, highlighting differences from traditional software package naming and presenting findings on PTM naming practices. The survey results indicate a mismatch between engineers' preferences and current practices in PTM naming. We then introduce DARA, the first automated DNN ARchitecture Assessment technique designed to detect PTM naming inconsistencies. Our results demonstrate that architectural information alone is sufficient to detect these inconsistencies, achieving an accuracy of 94% in identifying model types and promising performance (over 70%) in other architectural metadata as well. We also highlight potential use cases for automated naming tools, such as model validation, PTM metadata generation and verification, and plagiarism detection. Our study provides a foundation for automating naming inconsistency detection. Finally, we envision future work focusing on automated tools for standardizing package naming, improving model selection and reuse, and strengthening the security of the PTM supply chain.

Introduction: The "Wild West" of AI Model Naming

In the world of traditional software, naming a package is relatively straightforward—think requests for Python or express for Node.js. However, in the rapidly expanding universe of Pre-Trained Models (PTMs), naming conventions are far more complex and less standardized. A name like ast-finetuned-audioset-10-10-0.4593 is common, yet difficult for users to parse or verify.

Comparison of naming conventions across NPM, PyPI, and Hugging Face registries.

As deep learning becomes modular, engineers treat PTMs as components to be reused. Unlike traditional software libraries that describe what they do, PTM names often struggle to describe how they are built (architecture) and what they are for (task). This study explores the significant gap between how models are named and how engineers actually want to find them.

The Problem: Metadata Inconsistency

Currently, PTM naming is a manual, subjective process. Inaccurate or misleading names—such as a model labeled "ResNet" that is actually a "VGGNet"—hinder searchability, reduce trust, and create security risks like "typosquatting" or the accidental use of models with architectural backdoors.

An example of metadata inconsistency in a PTM package and its impact on the model lifecycle.

Engineers currently identify these naming inconsistencies through costly manual labor: reading model cards, visualizing architecture, or running the model on sample data to see if it behaves as expected. There is a clear need for automated tools to verify that a model's name matches its internal reality.

Methodology

The researchers employed a three-pronged approach:

  1. User Survey: 108 Hugging Face users were surveyed to understand their preferences and how they identify naming errors.
  2. Large-Scale Repository Mining: Using GPT-4, the team analyzed 14,296 PTM names from the PeaTMOSS dataset to extract naming elements like architecture, size, and dataset.
  3. DARA (DNN ARchitecture Assessment): The authors developed DARA, a tool that bypasses metadata entirely. DARA takes a model's raw weights, generates a computational graph, and uses machine learning to predict what the model's name should be based on its structure.

Key Insights and Results

  • Models as Hardware Specs: Users view PTM names less like software identifiers and more like hardware specification sheets, wanting names to encode both architecture and task.
  • The Preference Gap: While 60% of surveyed engineers prefer names combining implementation and task details, 59% of real-world models only include implementation details.
  • Architecture is Identity: A model's weight file is its "incontrovertible fact." Metadata can be faked, but the mathematical structure of the layers rarely lies.
  • DARA Performance: The tool is highly effective, identifying the broad model_type with 99% accuracy and specific architecture with 72% accuracy. Furthermore, the N-gram version of DARA is incredibly fast, processing a model in just 0.19 ms.

Future Directions

While DARA currently relies on architectural features, future work aims to incorporate weight analysis to detect more subtle differences, such as models sharing the same structure but trained on different datasets. The authors envision "automated linters" for model names and security applications to detect plagiarism or hidden architectural backdoors in the PTM supply chain.

Summary updated

Abstract

As innovation in deep learning continues, many engineers are incorporating Pre-Trained Models (PTMs) as components in computer systems. Some PTMs are foundation models, and others are fine-tuned variations adapted to different needs. When these PTMs are named well, it facilitates model discovery and reuse. However, prior research has shown that model names are not always well chosen and can sometimes be inaccurate and misleading. The naming practices for PTM packages have not been systematically studied, which hampers engineers' ability to efficiently search for and reliably reuse these models. In this paper, we conduct the first empirical investigation of PTM naming practices in the Hugging Face PTM registry. We begin by reporting on a survey of 108 Hugging Face users, highlighting differences from traditional software package naming and presenting findings on PTM naming practices. The survey results indicate a mismatch between engineers' preferences and current practices in PTM naming. We then introduce DARA, the first automated DNN ARchitecture Assessment technique designed to detect PTM naming inconsistencies. Our results demonstrate that architectural information alone is sufficient to detect these inconsistencies, achieving an accuracy of 94% in identifying model types and promising performance (over 70%) in other architectural metadata as well. We also highlight potential use cases for automated naming tools, such as model validation, PTM metadata generation and verification, and plagiarism detection. Our study provides a foundation for automating naming inconsistency detection. Finally, we envision future work focusing on automated tools for standardizing package naming, improving model selection and reuse, and strengthening the security of the PTM supply chain.
BibTeX
@article{jiang2025i,
  author = {Wenxin Jiang and Chingwo Cheung and George K. Thiruvathukal and James C. Davis and Thiruvathukal, George K. and Davis, James C.},
  title = {{"I see models being a whole other thing": An Empirical Study of Pre-Trained Model Naming Conventions and A Tool for Enhancing Naming Consistency}},
  journal = {Empir. Softw. eng},
  year = {2025},
  doi = {10.1007/s10664-025-10711-4},
}
Figure