TL;DR
This paper introduces a novel, static approach to pinpointing faults within system-level test code using Large Language Models (LLMs). By first estimating the execution trace of a failing test case and then using this pruned trace to guide an LLM, the method achieves high accuracy in fault localization while significantly reducing computational costs and inference time compared to existing techniques.
What Problem Does This Paper Address?
Debugging is a critical part of software development, and fault localization (FL) is the first step in identifying the root cause of failures. While much research has focused on FL for the system-under-test (SUT), faults in the system-level test code itself are often overlooked. These faulty test scripts can lead to misleading quality assessments and wasted debugging effort, especially in complex system-level testing scenarios. Traditional FL methods often rely on repeated test executions, which can be impractical due to high costs or non-deterministic failures. Existing LLM-based FL approaches primarily target the SUT, leaving a gap in automated, efficient FL for system-level test code.
Methods
The paper proposes a two-phase approach:
- Execution Trace Estimation: This phase is entirely static and does not require executing the test case. It involves:
- Source-to-Log Matching: Matching log messages from the execution log to corresponding log statements in the test code to identify executed lines.
- Estimation Algorithms: Three algorithms are proposed to infer the execution states of remaining lines:
- Fill-in-the-Gaps (T1): A simple algorithm assuming sequential execution, refined by considering known executed/non-executed statements.
- CFG-Based (T2): Uses Control Flow Graphs (CFGs) to identify potentially executed paths, accounting for branches.
-
Call Site Refinement (CSR): Removes function definitions that are not called within the estimated trace. These algorithms can be used individually or in combination to create a pruned execution trace.
-
LLM-Based Fault Localization: The pruned test code, along with the error message, is fed into an LLM using a carefully crafted prompt template. This template guides the LLM to identify and rank the most suspicious code elements (functions, blocks, or lines) responsible for the failure.
Datasets
The study uses a newly curated industrial dataset of 785 faulty Python system test cases provided by an industrial partner. This dataset is described as differing from typical benchmarks used for SUTFL due to its focus on test code faults and its industrial origin.
Key Insights and Ideas
- Test code faults are a significant problem: The paper highlights that faulty test code is a common issue, leading to misleading results and wasted debugging effort.
- Static analysis is feasible for FL: By cleverly estimating execution traces, the need for costly and often impractical repeated test executions can be avoided.
- Pruning the search space is crucial for LLMs: Providing LLMs with only relevant code snippets (based on estimated execution traces) significantly reduces token usage and inference time without sacrificing accuracy.
- Block-level granularity offers a good trade-off: For system-level test code, block-level FL provides a balance between the detail of line-level analysis and the broader context of function-level analysis.
- LLMs can be powerful for TCFL: The study demonstrates that LLMs, when properly guided, can effectively localize faults in complex test code.
Key Results
- High Accuracy: The best estimated trace (CSR(T1,dbt=EXE ∩ T2)) achieved a Masked Prediction F1 score (MPF1) of nearly 90%, indicating high accuracy in estimating execution states.
- Significant Pruning: The CSR(T1,dbt=NEXE ∩ T2) trace achieved the highest pruning rate (35.7%), reducing the search space considerably.
- Efficiency Gains: Pruning the test code reduced LLM inference time by up to 34% and token usage by up to 31% compared to using the full test code.
- Practical Setting: Top-3 block-level FL using the Tmin trace (CSR(T1,dbt=EXE ∩ T2)) was identified as a pragmatic setting, achieving 34% precision, 60.9% recall, and an 81% hit rate.
- Competitive Performance: The proposed TCFL method outperformed or matched state-of-the-art SUTFL baselines (FauxPy, IRFL, FlexFL) in terms of accuracy and was significantly more efficient.
- LLM Efficiency: The study showed that using smaller LLMs or more efficient models (like Qwen2.5-Coder-32B) can further reduce computational costs.
Limitations and Future Work
- Dataset Specificity: The study was conducted on Python test cases from a single industrial partner. Generalizability to other languages and domains needs further investigation.
- Ground Truth Limitations: The diff-based method for identifying faulty lines might include refactoring changes. While a 3-σ rule and manual inspection were used, some false positives might remain.
- Trace Estimation Assumptions: The Fill-in-the-Gaps algorithm assumes no branches, and the CFG-based algorithm has node-level granularity, which can lead to overestimation.
- Future Work:
- Enhancing trace estimation by incorporating more information from call sites and data flow graphs.
- Investigating the impact of syntactic and semantic validity of pruned test code on LLM performance.
- Exploring applications of estimated traces beyond FL, such as test code repair.
- Examining the effect of execution order for further pruning.
- Evaluating the approach on more diverse datasets.
Summary updated