Predicting Crash Fault Residence via Simplified Deep Forest Based on A Reduced Feature Set

Kunsong Zhao, Jin Liu, Zhou Xu, Li Li, Meng Yan, Jiaojiao Yu, Yuxuan Zhou

Abstract

Locating software bugs after a crash is a time-intensive task for developers. A critical first step is determining whether the fault resides within the generated stack trace (InTrace) or elsewhere (OutTrace). This paper introduces ConDF, a two-stage framework that combines consistency-based feature selection with a simplified Deep Forest architecture. Experimental results across seven major open-source Java projects demonstrate that ConDF significantly outperforms 17 baseline machine learning models, achieving superior accuracy and reliability in crash fault residence prediction.

Introduction

When a software system fails, it typically produces a stack trace—a snapshot of active functions at the moment of the crash. For developers, the immediate challenge is identifying the fault's location. If the bug is "InTrace," the search space is confined to the files listed in the stack trace. Conversely, an "OutTrace" bug requires a much broader search through the function invocation graph, consuming significant time and resources.

Existing approaches to crash fault residence prediction often rely on traditional machine learning models that struggle with complex data patterns or fail to optimize the feature set. This paper addresses these gaps by proposing the ConDF framework, which leverages advanced feature engineering and a layered ensemble model to improve prediction performance.

Methodology: The ConDF Framework

The ConDF framework consists of two primary stages: feature subset selection and model construction.

The ConDF framework architecture, illustrating the feature selection and cascade forest stages.

1. Consistency-based Feature Selection (Con)

Starting with 89 features extracted from stack traces and source code, the authors apply a consistency-based selection method. This algorithm identifies the smallest subset of features that retains the same discriminatory power as the original set. By filtering out noise and redundant metrics (such as lines of code or parameter counts), the feature set is reduced from 89 down to an average of just 6 to 8 highly representative features.

2. Simplified Deep Forest (DF)

The classification component utilizes a Deep Forest architecture, a decision tree ensemble that mimics the layered structure of deep learning. Unlike standard Deep Forest models designed for high-dimensional data (like images), ConDF uses a simplified version that omits multi-grained scanning and focuses on the cascade structure.

In this cascade, each level of the forest passes its class distribution predictions to the subsequent level, concatenated with the original features. This iterative process enables "representation learning," allowing the model to refine its understanding of the data layer by layer.

Experimental Evaluation

The framework was evaluated using a benchmark of seven prominent Java projects, including Apache Commons (Codec, Collections, IO), Jsoup, JSqlParser, Mango, and Ormlite-Core. To simulate realistic crash scenarios, the researchers employed mutation testing via the PIT tool, injecting single-point faults to generate InTrace and OutTrace crash instances.

ConDF's performance was compared against 17 baseline models, including traditional classifiers (SVM, Logistic Regression) and ensemble methods (XGBoost, AdaBoost, Bagging).

Results and Discussion

ConDF demonstrated superior performance across all evaluated metrics. It achieved an average F-measure of 0.722 for InTrace instances and 0.942 for OutTrace instances. Notably, in terms of the Matthews Correlation Coefficient (MCC)—a balanced measure for binary classification—ConDF scored 0.681, significantly outperforming the state-of-the-art "CraTer" model (0.446).

Statistical ranking of ConDF against baseline models across three performance metrics (F-InTrace, F-OutTrace, and MCC).

Key insights from the study include:

  • Feature Efficiency: Using only ~8% of the original features actually improved prediction performance, indicating that many traditional software metrics act as noise in crash analysis.
  • Model Robustness: The Deep Forest's cascade structure proved highly effective for tabular crash data, providing the benefits of deep architectures without the high computational cost or hyperparameter tuning required by Deep Neural Networks (DNNs).
  • Operational Efficiency: Despite its layered approach, ConDF is computationally efficient, with training and classification taking approximately 20 seconds on a standard workstation.

Conclusion

The ConDF framework provides a robust and efficient solution for predicting crash fault residence. By combining targeted feature selection with a simplified Deep Forest architecture, it offers developers a reliable tool to streamline the debugging process. Future work will explore the framework's applicability to other programming languages like C++ and Python, as well as the integration of imbalance learning techniques to further enhance performance on real-world crash data.

Summary updated

BibTeX
@inproceedings{zhao2021predicting,
  author = {Kunsong Zhao and Jin Liu and Zhou Xu and Li Li and Meng Yan and Jiaojiao Yu and Yuxuan Zhou},
  title = {{Predicting Crash Fault Residence via Simplified Deep Forest Based on A Reduced Feature Set}},
  booktitle = {ICPC},
  year = {2021},
  doi = {10.1109/icpc52881.2021.00031},
}
Figure