Introduction
In the field of Software Engineering (SE) research, "Mining Software Repositories" (MSR) has become a cornerstone for understanding development patterns, bug-fixing processes, and developer collaboration. However, the utility of these studies depends heavily on the quality of the underlying data. GitHub, the primary source for MSR, is a heterogeneous environment containing everything from production-grade software to "toy" projects, student assignments, and tutorials.
This noise is particularly problematic in the rapidly expanding domain of Machine Learning (ML). While the barrier to entry for ML development is low, the requirements for "engineered" or production-ready ML software are high. To address this, researchers have introduced NICHE, a manually curated dataset of 572 Python-based ML projects, designed to serve as a high-quality benchmark for the emerging sub-field of Mining Machine Learning Repositories (MLR).
The Quality Proxy Problem
A common practice in MSR research is using a repository's "star" count as a proxy for its engineering quality or maturity. However, popularity does not always equate to professional rigor. In the context of ML, a repository might gain thousands of stars because it provides a popular tutorial or a collection of pre-trained models, even if the code itself lacks modular architecture, testing, or continuous integration.
If researchers include these low-quality projects in their studies, they risk drawing biased or inaccurate conclusions about how ML software is actually built and maintained. NICHE aims to provide a "gold standard" dataset that distinguishes truly engineered software from experimental or educational repositories.
Methodology: Building the NICHE Dataset
The creation of NICHE involved a rigorous two-stage process combining automated heuristics with intensive manual curation.
1. Automated Filtering
The researchers initially identified projects utilizing major Python ML libraries, specifically Theano, PyTorch, and TensorFlow. To ensure a baseline level of activity and relevance, they applied the following filters:
- Minimum of 100 stars.
- Minimum of 100 commits.
- At least one commit occurring after May 2020.
This process resulted in a candidate pool of 572 projects.
2. Manual Curation
To move beyond automated proxies, two researchers independently audited every project in the candidate pool. They evaluated each repository across eight dimensions of software engineering maturity:
- Architecture: Logical organization into modules.
- Community: Presence of multiple active contributors.
- Continuous Integration (CI): Use of automated build/test tools (e.g., Travis CI, CircleCI).
- Documentation: Comprehensive README files and clear docstrings.
- History: Evidence of project evolution over time.
- Issues: Use of GitHub Issues for tracking bugs and features.
- License: Presence of a clear legal framework.
- Unit Testing: Evidence of automated testing suites.
Disagreements were resolved through consultation with a third, senior researcher. This manual effort required approximately 123 person-hours, ensuring a high-fidelity classification.
Dataset Composition and Key Insights
The final NICHE dataset classifies the 572 projects into two categories:
- Engineered Projects: 441 (77%)
- Non-Engineered Projects: 131 (23%)
The analysis of this curated data yielded several critical insights into the nature of ML repositories:
The Popularity Trap
The data confirmed that stars are an unreliable indicator of engineering maturity. Notably, nearly half (47.5%) of the "engineered" projects had between 100 and 500 stars. This suggests that many high-quality, professionally maintained ML projects remain relatively "quiet" in terms of public popularity, while some non-engineered projects may have high star counts due to their educational value.
Commits and Code Volume
The researchers found that commit history and code volume are much stronger signals of engineering rigor than stars. There is a significant positive correlation between the number of commits and the likelihood of a project being labeled as "engineered." Furthermore, the study identified a threshold in Lines of Code (LOC): for projects with fewer than 1,000 LOC, non-engineered projects actually outnumbered engineered ones. Once a project exceeds the 1,000 LOC mark, it is significantly more likely to exhibit professional software engineering traits.
Conclusion and Future Work
NICHE provides a vital resource for researchers looking to study production-grade ML software. By offering a dataset labeled with high precision, it enables the training and evaluation of automated classifiers (such as the existing Reaper or PHANTOM tools) to better identify quality repositories at scale.
While the dataset is a snapshot in time and involves a degree of subjective judgment, the double-blind review process mitigates these concerns. Future research using NICHE will focus on identifying the specific development patterns and "pain points" that distinguish ML engineering from traditional software development, ultimately leading to better tools and practices for the ML community.
The complete NICHE dataset is publicly available on GitHub.
Summary updated