Using GGNN to recommend log statement level

Mingzhe Li, Jianrui Pei, Jin He, Kevin Song, Frank Che, Yongfeng Huang, Chitai WangarXiv 2019 · arXiv

Introduction

Choosing the appropriate log level (e.g., INFO vs. DEBUG) is a frequent challenge for software developers, accounting for over 25% of all logging-related code changes. This research introduces a Gated Graph Neural Network (GGNN) approach that treats source code as a structured graph rather than sequential text, significantly improving the accuracy of log level recommendations compared to traditional NLP models.

Methodology: Code as a Program Graph

The primary innovation of this work is the transition from treating code as natural language to representing it as a Program Graph. This representation utilizes seven types of nodes (including tokens, identifiers, and AST elements) and 14 types of edges that capture complex relationships such as "Last Write," "Computed From," and "Guarded By."

The GGNN-based architecture for log level prediction, transforming source files into subgraphs for classification.

The model architecture consists of two main components:

  1. GGNN: A Gated Graph Neural Network uses Gated Recurrent Units (GRUs) to propagate information between nodes, allowing each node's representation to incorporate its structural context.
  2. MLP: A Multi-Layer Perceptron classifies the final vector representation of the log statement into one of six levels: trace, debug, info, warn, error, or fatal.

To ensure unbiased training, the authors replace the target log statement with a semicolon and extract a centered subgraph for inference, preventing the model from simply reading the existing label.

Experimental Results and Insights

The model was evaluated using a dataset of 8,000 labeled log samples from 29 high-star Java projects on GitHub. The GGNN model achieved an AUC of 0.880 and an accuracy of 65.7% on projects seen during training, significantly outperforming traditional sequential LSTM models (0.732 AUC).

Normalized confusion matrix showing the model's performance and the tendency toward adjacent-level errors.

A key insight from the results is the "Adjacent Error" phenomenon. As shown in the confusion matrix, when the model misclassifies a log level, it frequently selects a level adjacent to the ground truth (e.g., predicting INFO instead of DEBUG). This suggests the model successfully learns the underlying severity spectrum of the code, even when it misses the exact label.

Conclusion and Future Work

While the GGNN approach shows strong performance, challenges remain regarding project-specific logging standards and data sparsity. Future research will explore attention mechanisms to focus on critical graph components and apply similar graph-based architectures to other software engineering tasks like defect detection.

Summary updated

BibTeX
@misc{li2019using,
  author = {Mingzhe Li and Jianrui Pei and Jin He and Kevin Song and Frank Che and Yongfeng Huang and Chitai Wang},
  title = {{Using GGNN to recommend log statement level}},
  year = {2019},
  eprint = {1912.05097},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.05097},
}
Figure