This paper, presented at the 2022 Mining Software Repositories (MSR) conference as a Registered Report, proposes a novel way to triage the chaos of GitHub issue trackers using the mathematical concept of "surprisal."
TL;DR
Researchers are testing whether "surprisal"—an information-theoretic measure of how unexpected a text is—can predict which GitHub issues will be the hardest to resolve or the most important to the community. By training language models on thousands of repositories, they aim to provide developers with an automated "heads-up" for unusual events that require extra attention.
What Problem Does This Paper Address?
Software maintainers are often buried under a mountain of issues and pull requests (PRs). Identifying which of these are truly "notable"—either because they are unusually difficult to fix or because they represent critical, high-priority changes—is a manual, time-consuming process.
While previous tools have used simple code metrics (like lines of code changed) to flag unusual activity, this paper looks at the natural language within the issue tracker itself. The authors hypothesize that if an issue is written in a way that is statistically "surprising" compared to the project's typical language, it likely signals an underlying complexity or a significant event that developers should know about early.
Methods
The core of the study is the application of Information Theory to software engineering artifacts.
- Surprisal Calculation: The authors define surprisal as the negative logarithm of a word's probability given its context. To calculate this, they use a Statistical Language Model (SLM)—specifically a trigram model with Kneser-Ney smoothing.
- Cross-Entropy: To move from individual words to whole issues, they use cross-entropy. This measures how much the distribution of words in a specific issue deviates from the "true" distribution of words across the entire corpus of issues.
- Pre-processing: The text is cleaned by removing HTML, normalizing Unicode, and replacing code blocks with a special
[CODE]token to prevent unique variable names from skewing the "surprise" factor. - Correlation Analysis: The researchers plan to use multiple linear regression to see if high surprisal scores correlate with "Response Variables" like resolution time, number of participants, and community reactions.
Datasets
The study utilizes a massive dataset scraped from GitHub:
- The Top 5000: The 5,000 most "starred" repositories on GitHub.
- Filtering: To ensure data quality, they filter for English-language repositories and those with at least 1,000 issues.
- Scope: They analyze both Issues (bug reports/feature requests) and Pull Requests (proposed code changes) separately to see if surprisal behaves differently in more structured PR descriptions.
Key Insights and Ideas
- The "Naturalness" of Issues: The paper builds on the "naturalness" hypothesis—the idea that software-related text is repetitive and predictable. Surprisal is the tool used to find the "unnatural" exceptions.
- Defining "Difficulty" vs. "Importance": The authors provide a multi-faceted definition of these terms.
- Difficulty is measured by re-open rates, the volume of discussion, and the time to closure.
- Importance is measured by mentions in release notes, GitHub "reactions" (like 👍 or ❤️), and priority labels.
- The "Hiatus" Metric: One of the more creative ideas is the "Order of Address." By looking at which issues a developer chooses to work on first after a long break (a hiatus), the researchers can infer which issues the developer perceives as most important.
Key Results
Note: As a Registered Report, this paper outlines the experimental design and hypotheses; the full execution and quantitative results are intended for the follow-up phase of the study.
The authors' primary hypotheses are:
- H1: Statistical surprisal will align with human perceptions of what is "surprising."
- H2 & H3: High surprisal will correlate with higher resolution difficulty (more re-opens, more discussion, longer time to fix).
- H4 & H5: High surprisal will correlate with higher perceived importance (more reactions, priority labels, and inclusion in release notes).
Limitations and Future Work
- Model Complexity: The authors chose a trigram model for its simplicity and familiarity, but they acknowledge that modern neural language models (like Transformers) might capture context better, albeit at a higher computational cost.
- Causality: A high surprisal score might simply mean an issue is poorly written or contains a lot of jargon, rather than being inherently difficult. Disentangling "bad writing" from "complex problems" remains a challenge.
- Data Noise: GitHub stars are a proxy for popularity, not necessarily project health or documentation quality. Some high-star repos may have messy issue trackers that could provide noisy training data.
The ultimate goal is to integrate these surprisal metrics into developer dashboards, acting as an early-warning system for "bad surprises" before they become costly regressions.
Summary updated