Is Surprisal in Issue Trackers Actionable?

James Caddy, Markus Wagner, Christoph Treude, Earl T. Barr, Miltiadis AllamanisarXiv 2022 · DOI · arXiv

This paper, presented at the 2022 Mining Software Repositories (MSR) conference as a Registered Report, proposes a novel way to triage the chaos of GitHub issue trackers using the mathematical concept of "surprisal."

TL;DR

Researchers are testing whether "surprisal"—an information-theoretic measure of how unexpected a text is—can predict which GitHub issues will be the hardest to resolve or the most important to the community. By training language models on thousands of repositories, they aim to provide developers with an automated "heads-up" for unusual events that require extra attention.

What Problem Does This Paper Address?

Software maintainers are often buried under a mountain of issues and pull requests (PRs). Identifying which of these are truly "notable"—either because they are unusually difficult to fix or because they represent critical, high-priority changes—is a manual, time-consuming process.

While previous tools have used simple code metrics (like lines of code changed) to flag unusual activity, this paper looks at the natural language within the issue tracker itself. The authors hypothesize that if an issue is written in a way that is statistically "surprising" compared to the project's typical language, it likely signals an underlying complexity or a significant event that developers should know about early.

Methods

The core of the study is the application of Information Theory to software engineering artifacts.

  1. Surprisal Calculation: The authors define surprisal as the negative logarithm of a word's probability given its context. To calculate this, they use a Statistical Language Model (SLM)—specifically a trigram model with Kneser-Ney smoothing.
  2. Cross-Entropy: To move from individual words to whole issues, they use cross-entropy. This measures how much the distribution of words in a specific issue deviates from the "true" distribution of words across the entire corpus of issues.
  3. Pre-processing: The text is cleaned by removing HTML, normalizing Unicode, and replacing code blocks with a special [CODE] token to prevent unique variable names from skewing the "surprise" factor.
  4. Correlation Analysis: The researchers plan to use multiple linear regression to see if high surprisal scores correlate with "Response Variables" like resolution time, number of participants, and community reactions.

Datasets

The study utilizes a massive dataset scraped from GitHub:

  • The Top 5000: The 5,000 most "starred" repositories on GitHub.
  • Filtering: To ensure data quality, they filter for English-language repositories and those with at least 1,000 issues.
  • Scope: They analyze both Issues (bug reports/feature requests) and Pull Requests (proposed code changes) separately to see if surprisal behaves differently in more structured PR descriptions.

Key Insights and Ideas

  • The "Naturalness" of Issues: The paper builds on the "naturalness" hypothesis—the idea that software-related text is repetitive and predictable. Surprisal is the tool used to find the "unnatural" exceptions.
  • Defining "Difficulty" vs. "Importance": The authors provide a multi-faceted definition of these terms.
  • Difficulty is measured by re-open rates, the volume of discussion, and the time to closure.
  • Importance is measured by mentions in release notes, GitHub "reactions" (like 👍 or ❤️), and priority labels.
  • The "Hiatus" Metric: One of the more creative ideas is the "Order of Address." By looking at which issues a developer chooses to work on first after a long break (a hiatus), the researchers can infer which issues the developer perceives as most important.

Key Results

Note: As a Registered Report, this paper outlines the experimental design and hypotheses; the full execution and quantitative results are intended for the follow-up phase of the study.

The authors' primary hypotheses are:

  • H1: Statistical surprisal will align with human perceptions of what is "surprising."
  • H2 & H3: High surprisal will correlate with higher resolution difficulty (more re-opens, more discussion, longer time to fix).
  • H4 & H5: High surprisal will correlate with higher perceived importance (more reactions, priority labels, and inclusion in release notes).

Limitations and Future Work

  • Model Complexity: The authors chose a trigram model for its simplicity and familiarity, but they acknowledge that modern neural language models (like Transformers) might capture context better, albeit at a higher computational cost.
  • Causality: A high surprisal score might simply mean an issue is poorly written or contains a lot of jargon, rather than being inherently difficult. Disentangling "bad writing" from "complex problems" remains a challenge.
  • Data Noise: GitHub stars are a proxy for popularity, not necessarily project health or documentation quality. Some high-star repos may have messy issue trackers that could provide noisy training data.

The ultimate goal is to integrate these surprisal metrics into developer dashboards, acting as an early-warning system for "bad surprises" before they become costly regressions.

Summary updated

Abstract

Background. From information theory, surprisal is a measurement of how unexpected an event is. Statistical language models provide a probabilistic approximation of natural languages, and because surprisal is constructed with the probability of an event occuring, it is therefore possible to determine the surprisal associated with English sentences. The issues and pull requests of software repository issue trackers give insight into the development process and likely contain the surprising events of this process. Objective. Prior works have identified that unusual events in software repositories are of interest to developers, and use simple code metrics-based methods for detecting them. In this study we will propose a new method for unusual event detection in software repositories using surprisal. With the ability to find surprising issues and pull requests, we intend to further analyse them to determine if they actually hold importance in a repository, or if they pose a significant challenge to address. If it is possible to find bad surprises early, or before they cause additional troubles, it is plausible that effort, cost and time will be saved as a result. Method. After extracting the issues and pull requests from 5000 of the most popular software repositories on GitHub, we will train a language model to represent these issues. We will measure their perceived importance in the repository, measure their resolution difficulty using several analogues, measure the surprisal of each, and finally generate inferential statistics to describe any correlations.
BibTeX
@misc{caddy2022is,
  author = {James Caddy and Markus Wagner and Christoph Treude and Earl T. Barr and Miltiadis Allamanis},
  title = {{Is Surprisal in Issue Trackers Actionable?}},
  year = {2022},
  eprint = {2204.07363},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.07363},
}
Figure