Auditing a Model-Hub Benchmark: Data Quality, Training Budgets, and Side Information in HuggingBench

Michal PodstawskiarXiv 2026 · arXiv

TL;DR

This paper presents a rigorous audit of HuggingBench, a new benchmark for recommending resources on Hugging Face. The authors uncover critical data quality issues—most notably misaligned identifiers in knowledge graphs—and show that simple, text-based linear models can outperform complex state-of-the-art neural recommenders by 18% while training in minutes on a CPU.

What Problem Does This Paper Address?

As machine learning moves toward "model hubs" like Hugging Face, researchers are treating these collections as "model lakes" that require specialized recommendation systems to help users find relevant models, datasets, and Spaces. HuggingBench was recently introduced as the first major benchmark for this task.

However, benchmarks are "data products." If the files, evaluation protocols, or underlying data are flawed, those errors propagate silently, leading the community to draw incorrect conclusions about which algorithms actually work. This paper audits HuggingBench to ensure its results are what they claim to be and to see if any powerful signals in the data were left on the table.

The HuggingBench audit pipeline and identified issues, ranging from misaligned identifiers to truncated training budgets.

Methods

The authors performed an end-to-end audit using several techniques:

  • Executable Data Validation: They developed a suite of automated checks to verify internal consistency, such as "join validity"—ensuring that an ID for a model in one file actually refers to the same model in another.
  • Replication & Controlled Experiments: They replicated all 11 recommendation methods reported in the original HuggingBench study. They then ran controlled experiments, varying one factor at a time (e.g., training seeds, epoch budgets) to isolate the impact of data errors.
  • Strong Baselines: They introduced simple but powerful baselines, including EASE (Embarrassingly Shallow Autoencoders) and TF-IDF text representations, to provide a "sanity check" against the more complex graph neural networks (GNNs).

Datasets

The audit centers on HuggingBench, which is derived from HuggingKG, a massive crawl of the Hugging Face ecosystem (2.6 million nodes). The benchmark includes:

  • Interaction Data: Over 1 million user "likes."
  • Knowledge Graphs (KGs): Four external graphs (1-hop, 2-hop, publisher, and homogeneous) describing item relationships.
  • Side Information: Rich text from model/dataset cards and metadata.

The authors also cross-checked other popular recommendation datasets like Amazon-Book, Yelp2018, and MovieLens-1M to see if the identity issues they found were widespread.

Key Insights and Ideas

  • The Identity Crisis: In three of the four external KGs, item identifiers were misaligned. A recommender trying to learn from facts about "Model A" was actually being fed facts about an unrelated "Space B." This "silent failure" meant KG-based methods were being evaluated on essentially random noise.
  • Training on a Timer: The original benchmark used a strict 100-epoch budget. The audit revealed that many models were still significantly improving when they hit this limit. This artificial ceiling skewed the comparison between different methods.
  • Text is the Strongest Signal: Despite the complexity of the GNNs used in the original study, none of them utilized the actual text of the model cards. The audit found that this text is the most predictive feature available.
  • Metadata Shortcuts: For tasks like "lineage tracing" (identifying a model's parent), the answer is often explicitly written in the model card's text. A simple "card-reading" baseline can match the performance of complex knowledge graph embeddings.
Fig. 2. Validation MRR@5 (the early-stopping criterion) during training with a 500-epoch budget (seed 42); the dashed line marks the reported 100-epoch budget, dots the best epoch.

Key Results

  • Massive Gains from Data Fixes: Simply correcting the identifier misalignment improved every KG-based method, with gains of up to 73% in mean Recall@10.
  • Simple Beats Complex: A tuned linear model (EASE) using TF-IDF text features achieved a Recall@10 of 0.1846, outperforming the best reported neural result (0.1578) by 18%.
  • Efficiency: These high-performing linear models can be trained in under 6 minutes on a standard CPU, whereas the neural methods required hours or even days on high-end GPUs (like the RTX 3090/5090).
A summary of the issues found, how they were detected, and their quantitative effect on the benchmark results.

Limitations and Future Work

The authors acknowledge that "likes" are a relatively weak form of feedback compared to actual model downloads or fine-tuning activity. Additionally, their "inheritance cold-start" protocol is a simulation on a static snapshot of data; a more robust evaluation would follow models longitudinally as they are published.

For future benchmark creators, the paper suggests a "validate joins, not only files" approach and recommends always including simple, strong baselines that utilize all released side information to prevent researchers from chasing complexity where simplicity suffices.

Summary updated

Abstract

Benchmarks built from model hubs combine interaction data, knowledge graphs (KGs) and resource documentation, but valid files do not ensure that these artifacts describe the same objects. Treating the release as a data product, we audit HuggingBench (SIGIR 2025), a benchmark for recommending Hugging Face resources, through data validation, a replication of its eleven recommendation methods and controlled experiments with three training seeds. In three of its four external KGs, the files pass key and range checks but violate the item-entity identity constraint on which KG recommenders rely. Correcting this join improves every KG-based method in every seed, by up to 73% in mean Recall@10, and consistently reverses ten of 27 comparisons between KG configurations. The protocol matters as well: the 100-epoch budget ends before early stopping in seven to eight of 29 validation-selected settings, and 21 of 23 such setting-seed pairs improve with a longer budget. The released item text, which no reported method uses, is the strongest signal: closed-form linear models with text and the item graph exceed the best reported Recall@10 by 18% while fitting in minutes on a CPU. Checks of other recommendation releases show that individually valid releases of one dataset can use incompatible identifiers. We release corrected files, mappings, validators and logs, and derive practices for preserving identity and provenance across the artifacts of benchmarks and model lakes.
BibTeX
@misc{podstawski2026auditing,
  author = {Michal Podstawski},
  title = {{Auditing a Model-Hub Benchmark: Data Quality, Training Budgets, and Side Information in HuggingBench}},
  year = {2026},
  eprint = {2611.03727},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03727},
}
Figure