What makes a popular academic AI repository?

Yuanrui Fan, Xin Xia, David Lo, Ahmed E. Hassan, Shanping Li

TL;DR

In the world of AI research, a high-impact paper and a popular GitHub repository often go hand-in-hand. This paper analyzes over 1,000 academic AI repositories to identify the specific software engineering practices—such as detailed documentation, visual aids, and reproducibility tools—that differentiate "star" repositories from those that languish in obscurity.

What Problem Does This Paper Address?

Open science is the backbone of modern AI, with researchers increasingly expected to release code and data alongside their publications. However, simply "dumping" code on GitHub isn't enough; while some repositories garner thousands of stars, many others receive almost no attention.

The authors address a practical gap: What actually makes an academic AI repository popular? While previous studies have looked at general-purpose software popularity, academic repositories have unique requirements, such as the need to reproduce specific experimental results. By identifying the characteristics of successful repositories, the authors provide a roadmap for researchers to increase the visibility and impact of their work.

Methods

The researchers conducted an empirical study using a multi-step pipeline:

  1. Data Collection & Cleaning: They started with 1,448 papers from top-tier conferences (NeurIPS, ICML, CVPR, etc.) and manually verified which GitHub repositories were actually maintained by the original authors (excluding third-party implementations).
  2. Feature Extraction: They proposed 21 features across three dimensions:
  3. Code: Lines of code, number of modules, programming language, and ML framework.
  4. Reproducibility: Presence of datasets, Docker files, pre-trained models, and shell scripts.
  5. Documentation: Number of images, lists, code blocks, and links in the README.
  6. Statistical Analysis: They used the Wilcoxon rank-sum test and Cliff’s delta to see which features statistically separated "popular" (top 20% by stars) from "unpopular" (bottom 70%) repositories.
  7. Machine Learning: A Random Forest classifier was trained to rank the importance of these features in predicting popularity.
  8. Topic Modeling: They used Latent Dirichlet Allocation (LDA) on paper abstracts to determine if the subject of the research influenced popularity more than the quality of the repository.

Datasets

  • Primary Dataset: 1,149 verified academic AI repositories linked to papers published between 2015 and 2018 in five major conferences: NeurIPS, ICML, CVPR, ICCV, and ECCV.
  • Metadata: Google Scholar citation counts were collected to analyze the correlation between paper impact and repository popularity.

Key Insights and Ideas

  • Citations and Stars are Linked: There is a strong correlation (Spearman’s $\rho > 0.7$) between a paper’s citations and its repository's stars. While correlation isn't causation, it suggests that a polished repository can be a force multiplier for research impact.
  • Practice Over Topic: Surprisingly, the topic of the research (e.g., Reinforcement Learning vs. Computer Vision) has very little impact on repository popularity. Software engineering practices matter much more than the specific AI sub-field.
  • The "Visual" README: Popular repositories are significantly more likely to include images in their READMEs. These usually fall into two categories: result images (showing the model's output) and mechanism images (architecture diagrams).
  • Framework Trends: Repositories using PyTorch were more likely to be popular than those using other frameworks. Conversely, repositories using MATLAB or those that failed to explicitly state their framework in the README were significantly less popular.

Key Results

  • Top Differentiators: The three most important features for distinguishing popular repositories are:
  • The number of links to other GitHub repositories in the README.
  • The number of images in the README.
  • The inclusion of a software license.
  • Significant Features: 11 out of the 21 studied features showed statistically significant differences between popular and unpopular groups. Popular repos generally have more code files, more modules (better organization), and provide pre-trained models and shell scripts to ease reproduction.
  • The "Unknown" Penalty: Unpopular repositories are much more likely to belong to the "Unknown" category for ML frameworks, meaning they didn't clearly state what tools were needed to run the code.

Limitations and Future Work

  • Causality: The study identifies associations but cannot definitively say if a good repository causes a paper to be highly cited, or if famous authors simply write better code.
  • Conference Scope: The data is limited to five top-tier AI conferences. The findings might differ for niche or emerging fields.
  • README-Centric: Most documentation features were extracted only from the root README.md. The authors note that repositories might have extensive documentation in other files or external wikis that weren't captured.
  • Future Direction: The authors suggest building automated tools based on these features to help researchers predict and improve the potential popularity of their repositories before release.

Summary updated

Abstract

Many AI researchers are publishing code, data and other resources that accompany their papers in GitHub repositories. In this paper, we refer to these repositories as academic AI repositories. Our preliminary study shows that highly cited papers are more likely to have popular academic AI repositories (and vice versa). Hence, in this study, we perform an empirical study on academic AI repositories to highlight good software engineering practices of popular academic AI repositories for AI researchers. We collect 1,149 academic AI repositories, in which we label the top 20% repositories that have the most number of stars as popular, and we label the bottom 70% repositories as unpopular. The remaining 10% repositories are set as a gap between popular and unpopular academic AI repositories. We propose 21 features to characterize the software engineering practices of academic AI repositories. Our experimental results show that popular and unpopular academic AI repositories are statistically significantly different in 11 of the studied features—indicating that the two groups of repositories have significantly different software engineering practices. Furthermore, we find that the number of links to other GitHub repositories in the README file, the number of images in the README file and the inclusion of a license are the most important features for differentiating the two groups of academic AI repositories. Our dataset and code are made publicly available to share with the community.
BibTeX
@inproceedings{fan2021what,
  author = {Yuanrui Fan and Xin Xia and David Lo and Ahmed E. Hassan and Shanping Li},
  title = {{What makes a popular academic AI repository?}},
  booktitle = {Empir. Softw. Eng},
  year = {2021},
  url = {https://openreview.net/forum?id=1fHKdldPYi},
}

Loading related content...

Figure