Abstract
Many AI researchers are publishing code, data and other resources that accompany their papers in GitHub repositories. In this paper, we refer to these repositories as academic AI repositories. Our preliminary study shows that highly cited papers are more likely to have popular academic AI repositories (and vice versa). Hence, in this study, we perform an empirical study on academic AI repositories to highlight good software engineering practices of popular academic AI repositories for AI researchers. We collect 1,149 academic AI repositories, in which we label the top 20% repositories that have the most number of stars as popular, and we label the bottom 70% repositories as unpopular. The remaining 10% repositories are set as a gap between popular and unpopular academic AI repositories. We propose 21 features to characterize the software engineering practices of academic AI repositories. Our experimental results show that popular and unpopular academic AI repositories are statistically significantly different in 11 of the studied features---indicating that the two groups of repositories have significantly different software engineering practices. Furthermore, we find that the number of links to other GitHub repositories in the README file, the number of images in the README file and the inclusion of a license are the most important features for differentiating the two groups of academic AI repositories. Our dataset and code are made publicly available to share with the community.
Introduction
In the world of AI research, a high-impact paper and a popular GitHub repository often go hand-in-hand. Open science is the backbone of modern AI, with researchers increasingly expected to release code and data alongside their publications. However, simply "dumping" code on GitHub is rarely sufficient for impact. While some repositories garner thousands of stars, many others receive almost no attention. This paper addresses a practical gap: What actually makes an academic AI repository popular? By identifying the characteristics of successful repositories, the authors provide a roadmap for researchers to increase the visibility and impact of their work.
Methodology and Data
The researchers conducted an empirical study using a multi-step pipeline starting with 1,448 papers from top-tier conferences (NeurIPS, ICML, CVPR, ICCV, and ECCV). After manual verification, 1,149 repositories maintained by the original authors were selected for analysis.
To differentiate between levels of success, the study categorized repositories based on their star counts. As shown in the distribution below, the authors labeled the top 20% of repositories as "popular" and the bottom 70% as "unpopular," creating a clear statistical gap between the two groups.
The researchers proposed 21 features across three dimensions to characterize these repositories:
- Code: Lines of code, number of modules, programming language, and ML framework.
- Reproducibility: Presence of datasets, Docker files, pre-trained models, and shell scripts.
- Documentation: Number of images, lists, code blocks, and links in the README.
Statistical analysis was performed using the Wilcoxon rank-sum test and Cliff’s delta, followed by training a Random Forest classifier to rank the importance of these features in predicting popularity.
Key Insights and Results
Citations and Stars are Linked
There is a strong correlation (Spearman’s $\rho > 0.7$) between a paper’s citations and its repository's stars. While correlation isn't causation, it suggests that a polished repository can be a force multiplier for research impact.
Practice Over Topic
Surprisingly, the topic of the research (e.g., Reinforcement Learning vs. Computer Vision) has very little impact on repository popularity. Software engineering practices matter much more than the specific AI sub-field. Popular repositories generally have more code files, better organization (more modules), and provide pre-trained models and shell scripts to ease reproduction.
The "Visual" README
Popular repositories are significantly more likely to include images in their READMEs. These usually fall into two categories: result images (showing the model's output) and mechanism images (architecture diagrams). In fact, the three most important features for distinguishing popular repositories are:
- The number of links to other GitHub repositories in the README.
- The number of images in the README.
- The inclusion of a software license.
Framework Trends
Repositories using PyTorch were more likely to be popular than those using other frameworks. Conversely, repositories using MATLAB or those that failed to explicitly state their framework in the README (the "Unknown" category) were significantly less popular.
Limitations and Future Work
While the study identifies strong associations, it cannot definitively prove causality—it is possible that famous authors simply write better code or have more resources for documentation. Additionally, the data is limited to five top-tier AI conferences and primarily focuses on the root README.md file. Future work may involve building automated tools to help researchers predict and improve the potential popularity of their repositories before release.
Conclusion
For AI researchers, the repository is as much a part of the scientific contribution as the paper itself. By adopting better software engineering practices—specifically by providing visual documentation, clear framework requirements, and reproducibility tools—researchers can significantly increase the reach and impact of their work.
Summary updated