The Power of Types: Exploring the Impact of Type Checking on Neural Bug Detection in Dynamically Typed Languages

Boqi Chen, José Antonio Hernández López, Gunter Mussbacher, Dániel Varró

Introduction

Neural Bug Detectors (NBDs) have emerged as powerful tools for identifying errors in source code. However, in dynamically typed languages like Python, these models often inadvertently focus on "low-hanging fruit"—simple type errors that traditional static analysis tools could easily identify. This research investigates how the presence of these "easy" bugs in training data biases NBD performance and proposes a pipeline approach to force models toward more complex logic errors.

Methodology

The researchers evaluated five state-of-the-art neural architectures: GGNN (Graph-based), GREAT (Hybrid), and three Transformer-based models (CodeBERT, GraphCodeBERT, and UniXcoder). The core of their approach is a multi-stage pipeline where a program is first validated by a static type checker (such as mypy or pytype). Only programs that pass this initial check are then analyzed by the NBD.

The proposed bug detection pipeline integrating a type checker and a neural bug detector.

To further refine model performance, the authors experimented with "Filtered" training datasets, where type-related bugs were removed and replaced with other bug types via oversampling. This intervention aims to prevent the model from relying on simple type inconsistencies as proxies for bugs.

Dataset Analysis and Type Bias

The study utilized four primary datasets focused on variable misuse bugs: Synthetic 1 (ETH Py-150), Synthetic 2 (an open-source training set), Real (merged Real-Py150 & Real-PyPI), and a manually Annotated subset of the Real dataset.

Distribution of bug types across synthetic and real-world datasets.

Analysis of these datasets revealed a significant "type bias." In synthetic datasets, type-related bugs (like name errors and attribute errors) account for nearly 20% of the data. In unannotated real-world data, this figure is lower (approximately 5%), but still significant enough to influence model training. Training on these sets biases the model toward patterns that a simple static tool could solve more efficiently and explainably.

Key Results and Insights

The research yielded several critical insights into the relationship between types and neural bug detection:

  • The Redundancy Trap: Training NBDs on unrefined data leads to models that primarily catch errors already solvable by efficient, explainable static tools. If a developer is already using a tool like mypy, an NBD that primarily catches these same errors is redundant.
  • Pipeline Synergy: Integrating a type checker into the detection pipeline significantly improves utility. On type-annotated programs, the pipeline approach increased precision by an average of 5.73% and recall by 8.78% compared to a standalone NBD. The type checker handles the "how" (data structures), leaving the NBD to handle the "what" (logic).
  • Improved Focus through Filtering: Training on filtered data—where simple type errors are removed—forces the neural network to learn more subtle, non-trivial logic patterns. This led to improved F-1.5 scores (prioritizing recall) in 9 out of 10 experimental cases. For instance, the GREAT model's F-1.5 score on real bugs improved from 15.98% to 17.82% after filtering.

Conclusion and Future Work

By explicitly accounting for type information, developers can create NBDs that are more aligned with realistic development workflows. While this study focused on variable misuse in Python, the principles of filtering "easy" bugs to improve model focus likely apply across various bug categories and dynamic languages.

Future work will explore expanding this methodology to other error types, such as incorrect binary operators or return values. Additionally, the authors suggest investigating how neural type inference tools could be integrated into the NBD training pipeline to further boost performance in unannotated codebases.

Summary updated

BibTeX
@inproceedings{chen2025power,
  author = {Boqi Chen and José Antonio Hernández López and Gunter Mussbacher and Dániel Varró},
  title = {{The Power of Types: Exploring the Impact of Type Checking on Neural Bug Detection in Dynamically Typed Languages}},
  booktitle = {ICSE},
  year = {2025},
  doi = {10.1109/icse55347.2025.00088},
}
Figure