Exploring Naming Conventions (and Defects) of Pre-trained Deep Learning Models in Hugging Face and Other Model Hubs

Wenxin Jiang, Chingwo Cheung, George K. Thiruvathukal, James C. DavisarXiv 2023 · arXiv · OpenReview

TL;DR

This paper investigates how Pre-Trained Models (PTMs) are named in popular hubs like Hugging Face, finding that PTM names are more descriptive than traditional software package names but still suffer from inconsistencies. The authors developed a tool called DARA to automatically detect these naming defects, which can hinder model discovery and reuse.

What Problem Does This Paper Address?

As deep learning models (PTMs) become increasingly integrated into software systems, their discoverability and reusability are crucial. Effective naming of these PTMs is key to this process, but prior research suggests that model names are often poorly chosen, inaccurate, or misleading. This lack of systematic study into PTM naming conventions and defects hinders engineers' ability to efficiently search for and reliably reuse these models. The paper aims to fill this gap by empirically studying PTM naming practices and developing a tool to identify naming inconsistencies.

Methods

The study employs a mixed-methods approach.

  1. Survey Study: A survey of 108 Hugging Face users was conducted to understand their preferences for PTM naming elements and conventions, and how they identify naming inconsistencies.
  2. Repository Mining: A large-scale mining study of 14,296 PTM packages from Hugging Face was performed to analyze actual naming practices. This involved using a large language model (LLM) pipeline to extract naming elements and conventions.
  3. Automated Naming Assessment Tool (DARA): The authors developed DARA, an anomaly detection algorithm designed to identify naming inconsistencies by analyzing architectural information from PTMs. DARA converts computational graphs into abstract architectures, extracts features (n-grams, CNNs, transformers), and uses a classifier to detect mismatches between the model's architecture and its metadata.

Datasets

  • PeaTMOSS dataset: Used for repository mining and evaluation of DARA. This dataset contains PTM packages with over 50 downloads from Hugging Face.
  • Hugging Face registry: The primary source for PTM packages analyzed in the study.

Key Insights and Ideas

  • PTMs are "literature" for humans: The paper highlights a quote by Donald Knuth, emphasizing that PTM names, like literature, are primarily for human understanding, not just machine processing. This underscores the importance of clear and informative naming.
  • PTM names are richer than traditional software names: Unlike traditional software packages that might prioritize brevity or branding, PTM names often encode detailed architectural and functional information, acting more like technical specifications.
  • A gap exists between user preference and practice: While users prefer PTM names that combine implementation details with application purpose, actual PTM packages often rely solely on implementation units.
  • Architectural information is key, but not always sufficient: DARA's success in detecting model_type inconsistencies (99% accuracy) suggests architectural features are powerful for broad categorization. However, distinguishing between similar architectures or tasks often requires more nuanced information.
  • Manual verification is common but costly: Engineers currently rely on reading model cards, external information, or inspecting the PTM itself to identify naming issues, which is time-consuming and error-prone.

Key Results

  • RQ1 (PTM vs. Traditional Naming): 88% of respondents found PTM naming to be different from traditional package naming, primarily due to greater semantic knowledge embedded in PTM names.
  • RQ2 (Naming Elements): Survey participants preferred naming by architecture/function. Mining results showed that while architecture is widely included, task and versioning details are less common in practice than preferred.
  • RQ3 (Identifying Inconsistencies): Users primarily rely on manual metadata validation (69%) and model inspection (46%) to identify naming inconsistencies.
  • RQ4 (DARA Performance):
    • model_type detection: DARA achieved up to 99.0% accuracy with transformer-based models.
  • architecture classification: The best transformer-based models achieved up to 72.0% accuracy.
  • task classification: N-gram based models outperformed transformers, achieving up to 76.3% accuracy (Top-1).
  • Efficiency: N-gram + MLP classifiers offered the lowest latency (0.19 ms/PTM) and highest throughput (>28,000 PTMs/s), making them suitable for real-time applications. Transformer-based models are slower but more accurate.

Limitations and Future Work

  • Limitations:
    • The study acknowledges potential bias in the survey due to targeting experienced users.
  • The repository mining excluded less popular packages, potentially missing insights from those models.
  • DARA's reliance on architectural information alone is insufficient for fine-grained classification tasks like distinguishing between similar architectures or tasks.
  • The dataset used for DARA evaluation was imbalanced, with fewer multimodal and audio models.
  • The study focused on data from Hugging Face, and findings might not generalize perfectly to other registries.
  • Future Work:
    • Developing automated tools for standardizing PTM package naming.
  • Improving model selection and reuse through better naming and metadata.
  • Enhancing PTM supply chain security by leveraging naming information to detect malicious practices.
  • Exploring more advanced feature extraction techniques (e.g., incorporating training dynamics, contrastive learning) to improve DARA's accuracy for architecture and task classification.
  • Investigating how naming convention preferences and practices vary between experts and novices.
  • Developing linters for package names, similar to those for code.
  • Exploring automated name generation based on static analysis.
  • Extending DARA to incorporate more meta-features for enhanced AutoML processes.

Summary updated

Abstract

As innovation in deep learning continues, many engineers want to adopt Pre-Trained deep learning Models (PTMs) as components in computer systems. PTMs are part of a research-to-practice pipeline: researchers publish PTMs, which engineers adapt for quality or performance and then deploy. If PTM authors choose appropriate names for their PTMs, it could facilitate model discovery and reuse. However, prior research has reported that model names are not always well chosen, and are sometimes erroneous. The naming conventions and naming defects for PTM packages have not been systematically studied - understanding them will add to our knowledge of how the research-to-practice process works for PTM packages In this paper, we report the first study of PTM naming conventions and the associated PTM naming defects. We define the components of a PTM package name, comprising the package name and claimed architecture from the metadata. We present the first study focused on characterizing the nature of naming in PTM ecosystem. To this end, we developed a novel automated naming assessment technique that can automatically extract the semantic and syntactic patterns. To identify potential naming defects, we developed a novel algorithm, automated DNN ARchitecture Assessment pipeline (DARA), to cluster PTMs based on architectural differences. Our study suggests the naming conventions for PTMs, and frames the naming conventions as signal of the research-to-practice relationships in the PTM ecosystem. We envision future works on further empirical study on leveraging meta-features of PTMs to support model search and reuse.
BibTeX
@misc{jiang2023exploring,
  author = {Wenxin Jiang and Chingwo Cheung and George K. Thiruvathukal and James C. Davis},
  title = {{Exploring Naming Conventions (and Defects) of Pre-trained Deep Learning Models in Hugging Face and Other Model Hubs}},
  year = {2023},
  eprint = {2310.01642},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.01642},
}

Loading related content...

Figure