Code Revert Prediction with Graph Neural Networks: A Case Study at J.P. Morgan Chase

Yulong Pei, Salwa Alamir, Rares Dolga, Sameena ShahSdd@sigsoft fse 2023 · DOI

TL;DR

This paper introduces "code revert prediction," a novel task in software engineering focused on identifying code changes likely to be rolled back. Using J.P. Morgan Chase's real-world Python codebase, they demonstrate that Graph Neural Networks (GNNs) combined with techniques to handle imbalanced data (like downsampling) significantly outperform traditional methods for this critical task.

What Problem Does This Paper Address?

In software development, when a code change introduces a severe issue, the quickest fix is often to revert it to a previous stable version. This "code revert" is a strong indicator of a problematic change. While predicting code defects is a well-studied area, predicting reverts specifically, especially in large, regulated industrial environments, has been largely overlooked. Traditional defect prediction methods often struggle with the unique challenges of industrial settings, such as massive codebases and limited access to dynamic analysis tools. Furthermore, code commit data is highly imbalanced, with reverts being rare events, making standard classification approaches ineffective. This paper tackles the problem of predicting code reverts by leveraging code import relationships and GNNs, while specifically addressing the data imbalance challenge.

Methods

The core of the paper's methodology lies in representing code as a graph and applying Graph Neural Networks (GNNs).

  • Graph Construction: Instead of relying on Abstract Syntax Trees (ASTs) or Data Flow Graphs (DFGs) which often require dynamic analysis (infeasible in their industrial setting), they construct a static "code import graph." Each node represents a Python script, and an edge connects two nodes if one script imports the other. This captures code dependencies.
  • Representation Learning: They use GNNs (specifically mentioning GCNs) to learn representations of these code graphs. These learned representations capture both the structure of code dependencies and the code's inherent features.
  • Handling Imbalance: Recognizing that reverts are rare (less than 4% of commits), they explore three main strategies:
  • Imbalance Classification: Using techniques like upsampling, downsampling, and SMOTE (Synthetic Minority Over-sampling Technique) to balance the dataset before training a classifier.
  • Anomaly Detection: Treating reverts as anomalies within the dataset and using anomaly detection algorithms.
  • Combined Strategy (Strategy 3): Applying imbalance handling techniques before constructing the graph and feeding it to a GNN. They also compare specialized GNNs for anomaly detection (Dominant) and imbalance classification (GraphSMOTE).

Datasets

The study uses a real-world dataset of Python code commit data from J.P. Morgan Chase. This dataset comprises over 10 million lines of code from a single codebase, collected over one month, with approximately 30,000 commits. A key characteristic of this dataset is its extreme imbalance, with less than 4% of commits resulting in a revert. No new benchmark dataset was introduced; rather, they utilized their internal, real-world data.

Key Insights and Ideas

  • Code Revert Prediction is a Distinct Problem: The paper establishes code revert prediction as a unique and important task, different from general code defect prediction, with significant practical implications for industrial software development.
  • Graph Structure Matters: Code import relationships, when modeled as a graph, provide valuable information for predicting reverts. GNNs are well-suited to leverage this structural information.
  • Imbalance is the Dominant Challenge: Traditional classifiers and even anomaly detection methods struggle significantly with the highly imbalanced nature of revert data. Imbalance handling techniques are crucial for achieving meaningful performance.
  • Downsampling + GCN is a Strong Baseline: Combining simple downsampling with a standard GCN model proved to be a highly effective strategy, often outperforming more complex specialized GNNs for imbalance or anomaly detection. This suggests that careful data preprocessing can be as impactful as advanced model architectures.
  • Industrial Constraints Shape Methodology: The need to avoid dynamic analysis due to industry regulations and scale significantly influenced the choice of graph construction methods (static import graphs).

Key Results

  • Downsampling + GCN Achieves Best Performance: In their experiments, the combination of downsampling (to address class imbalance) and a Graph Convolutional Network (GCN) achieved the highest performance across both AUC-ROC and Macro F1 metrics. For instance, Downsampling + GCN achieved an AUC-ROC of 0.7269 and a Macro F1 of 0.5695.
  • Traditional Classifiers Fail: Standard classifiers like Logistic Regression (LR), SVM, and Random Forest performed poorly, often failing to identify any reverts, especially when dealing with the imbalanced data without specific handling.
  • Imbalance Methods Outperform Anomaly Detection: Strategies focused on handling class imbalance (upsampling, downsampling, SMOTE) generally yielded better results than anomaly detection techniques for this specific problem.
  • node2vec + Raw Features Show Promise: For Strategies 1 and 2 (without Strategy 3's pre-processing), combining node2vec embeddings (for graph structure) with raw code features provided the best performance among those approaches.

Limitations and Future Work

  • Graph Construction Granularity: The authors acknowledge that their import graph is unidirectional and could be more fine-grained. Exploring directed graphs or incorporating other relationships (like push sets) could improve performance.
  • Data Imbalance: While they explored various methods, the extreme imbalance remains a significant challenge. Future work could focus on developing more sophisticated imbalance handling techniques tailored for code revert prediction.
  • Explainability: As with many GNN applications, understanding why a model predicts a revert is crucial for adoption. Future research should focus on enhancing the interpretability of these models.
  • Noisy Labels: A single commit can involve multiple scripts. If any script is reverted, the entire commit is labeled as a revert. This can introduce noise. Finer-grained revert labels for individual scripts would be beneficial.
  • Future Directions: The authors suggest exploring more advanced imbalance classification and explainability techniques, as well as developing novel methods specifically designed for code revert prediction beyond general anomaly detection.

Summary updated

BibTeX
@inproceedings{pei2023code,
  author = {Yulong Pei and Salwa Alamir and Rares Dolga and Sameena Shah},
  title = {{Code Revert Prediction with Graph Neural Networks: A Case Study at J.P. Morgan Chase}},
  booktitle = {Sdd@sigsoft fse},
  year = {2023},
  doi = {10.1145/3617572.3617879},
}
Figure