TL;DR
This paper provides a comprehensive empirical study of ChatGPT-related projects hosted on GitHub, categorizing them and analyzing the issues users encounter. The findings reveal distinct patterns in project types, issue topics, and their evolution, offering valuable insights for developers and platform managers to improve project development and management.
What Problem Does This Paper Address?
The rapid emergence of ChatGPT has led to a surge in related open-source projects on GitHub. However, the platform lacks detailed categorization for these projects, making it difficult for users to discover relevant ones. Furthermore, the issues reported by users are diverse, ranging from installation to usage and updates, and there's a need to understand their prevalence and nature to help developers prioritize and improve development efficiency. This paper aims to fill this gap by systematically analyzing ChatGPT-related projects and their associated issues on GitHub.
Methods
The researchers employed a multi-stage methodology:
- Project Collection and Selection: They scraped 71,244 projects from GitHub using the keyword "ChatGPT" and selected the top 200 most starred projects as their dataset.
- Project Categorization: Using an open card sorting approach, they manually categorized the 200 projects into three primary types: "ChatGPT Implementation & Training," "ChatGPT Application," and "ChatGPT Improvement & Extension," along with an "Other ChatGPT-related Project" category.
- Issue Collection and Preprocessing: They collected 23,609 issues from 118 projects within the three primary categories. A novel preprocessing pipeline was developed to handle issues written in both English and Chinese, including noise removal, word segmentation, lemmatization, stopword removal, and a unique Chinese-to-English word translation step for unified topic modeling.
- Topic Modeling: Latent Dirichlet Allocation (LDA) was used to identify ten distinct issue topics from the preprocessed issues.
- Analysis of Popularity, Difficulty, and Evolution:
- Popularity: Measured by the proportion of issues assigned to each topic within different project categories.
- Difficulty: Assessed using metrics like average comments, average participants, and closing rate of issues.
- Evolution: Analyzed using the Mann-Kendall trend test to identify increasing or decreasing trends in issue topic impact over time.
Datasets
- Primary Dataset: 200 representative ChatGPT-related projects from GitHub, selected based on the number of stars.
- Issue Dataset: 23,609 issues collected from 118 of these projects.
Key Insights and Ideas
- Project Categories Reflect Development Trends: The growth of "ChatGPT Application" and "ChatGPT Improvement & Extension" projects closely mirrors ChatGPT's own development milestones (e.g., GPT-4 release, iOS app launch). "ChatGPT Implementation & Training" projects surged with the rise of open-source large language models.
- Issue Topics Vary Significantly by Project Type: The popularity of issue topics differs greatly across the three main project categories. For instance, "model reply" is highly popular in "Implementation & Training" projects, while "interaction interface" dominates "Application" projects.
- Difficulty is Tied to Project Focus: "Model reply" issues are harder to resolve in "Implementation & Training" projects, likely due to the complexity of fine-tuning models. In contrast, "request method" issues are challenging in "Application" projects due to the diverse ways users interact with APIs.
- Evolutionary Trends Offer Predictive Power: Issue topics evolve over time, with some topics increasing (e.g., "npm runtime error" and "gpt conversation" in "Implementation & Training") and others decreasing (e.g., "docker deployment" and "login access" in "Application"). This suggests a shift in user focus as projects mature.
Key Results
- Project Categorization: Identified three primary categories: ChatGPT Implementation & Training (8 projects), ChatGPT Application (55 projects), and ChatGPT Improvement & Extension (55 projects).
- Issue Topics: Discovered ten distinct issue topics, including "request method," "model reply," "interaction interface," "project language," "docker deployment," "npm runtime error," "module import & result export," "gpt conversation," "login access," and "api key."
- Popularity Differences: Significant variations in topic popularity were observed across categories. For example, "model reply" was the most popular in "Implementation & Training" (22.1%), while "interaction interface" was most popular in "Application" (15.8%).
- Difficulty Insights: "Model reply" had the lowest closing rate (57.2%) in "Implementation & Training," indicating high difficulty. Conversely, "gpt conversation" had the lowest closing rate (61.9%) in "Improvement & Extension," suggesting challenges in improving conversational aspects.
- Evolutionary Trends: Identified specific trends for topics within each category. For instance, "model reply" decreased in "Implementation & Training," while "npm runtime error" and "gpt conversation" increased.
Limitations and Future Work
- Limitations:
- Subjectivity in Categorization: While inter-rater agreement was high, manual categorization inherently involves some subjectivity.
- Generalizability: The study focused on the top 200 most starred projects, which might not represent all ChatGPT-related projects.
- Future Work:
- Automated Issue Summarization: Generate summaries for issues within specific topics to help developers quickly grasp user concerns.
- Solution Recommendation: Synthesize existing solutions to similar issues to recommend candidate fixes for developers.
- Broader Platform Analysis: Extend the analysis to other project hosting platforms beyond GitHub.
Summary updated