Logic Error Localization in Student Programming Assignments Using Pseudocode and Graph Neural Networks

Zhenyu Xu, Kun Zhang, Victor S. Sheng

Abstract

Pseudocode is extensively used in introductory programming courses to instruct computer science students in algorithm design, utilizing natural language to define algorithmic behaviors. This learning approach enables students to convert pseudocode into source code and execute it to verify their algorithms' correctness. This process typically introduces two types of errors: syntax errors and logic errors. Syntax errors are often accompanied by compiler feedback, which helps students identify incorrect lines. In contrast, logic errors are more challenging because they do not trigger compiler errors and lack immediate diagnostic feedback, making them harder to detect and correct. To address this challenge, we developed a system designed to localize logic errors within student programming assignments at the line level. Our approach utilizes pseudocode as a scaffold to build a code-pseudocode graph, connecting symbols from the source code to their pseudocode counterparts. We then employ a graph neural network to both localize and suggest corrections for logic errors. Additionally, we have devised a method to efficiently gather logic-error-prone programs during the syntax error correction process and compile these into a dataset that includes single and multiple line logic errors, complete with indices of the erroneous lines. Our experimental results are promising, demonstrating a localization accuracy of 99.2% for logic errors within the top-10 suspected lines, highlighting the effectiveness of our approach in enhancing students' coding proficiency and error correction skills.

Introduction

In introductory computer science, students face two main hurdles: syntax errors and logic errors. Syntax errors are relatively easy to handle because compilers provide immediate, specific feedback (e.g., "missing semicolon on line 10"). Logic errors—often called semantic errors—are the "silent killers" of programming assignments. The code runs without crashing, but it produces the wrong output because the underlying algorithm is flawed.

Common types of logic errors addressed by the localization system.

Currently, localizing these errors is a manual, time-consuming process for both students and instructors. While automated tools exist for professional software, they often struggle with the small-scale, highly variable code found in student assignments. This research addresses the lack of automated, line-level diagnostic feedback for logic errors in educational settings by leveraging the student's own pseudocode as a formal specification.

Methodology

The proposed system utilizes a multi-layered deep learning architecture that treats pseudocode as a scaffold for the source code.

Architecture of the logic error localization system combining GNNs and CodeBERT.

The architecture consists of three primary components:

  1. The Code-Pseudocode Graph: The system builds a graph where nodes represent tokens from both the source code and the pseudocode. Edges connect matching tokens, creating a structural map between the student's intent and their implementation.
  2. Hybrid Neural Architecture:
    • BiLSTMs: Used to capture the sequential flow of lines in both languages.
  3. Graph Attention Layer (GAT): This component allows the model to weigh the importance of connections between code and pseudocode tokens, tracing the logical flow across the two representations.
  4. CodeBERT: A pre-trained model used to calculate "Semantic Alignment Scores," measuring how well a specific line of code matches its corresponding pseudocode description.
  5. Dynamic Training: The model uses a dual-focus loss function. Early in training, it focuses on simple error prediction. As it matures, it shifts focus toward semantic alignment, ensuring it understands the relationship between the code and the pseudocode's instructions.

Datasets

The researchers utilized the SPoC (Search-based Pseudocode to Code) dataset, which contains over 18,000 C++ programs with human-written pseudocode. To create a realistic error dataset, they "mined" for errors by observing the behavior of syntax-repair tools like DrRepair. When syntax is fixed but the program still fails test cases, a logic error is effectively isolated. This method was used to curate two specialized benchmarks: S-Logic-Err (single-line errors) and M-Logic-Err (multi-line errors).

Results

The proposed method was compared against classic Spectrum-Based Fault Localization (SBFL) tools like Tarantula, Ochiai, and DStar.

  • Top-10 Accuracy: On the single-line error dataset, the model achieved 99.2% accuracy, significantly outperforming DStar (78.3%).
  • Top-1 Accuracy: The model correctly identified the exact error line on the first try 36.1% of the time, nearly double the performance of traditional tools in this context.
  • Multi-line Performance: Even on the harder multi-line dataset, the model maintained a 96.4% top-10 accuracy, proving it can handle complex logical failures spanning several lines.

Conclusion

This research demonstrates that pseudocode can serve as a reliable map for localizing logic errors in educational programming tasks. By combining structural analysis via GNNs with semantic meaning from CodeBERT, the system provides high-accuracy diagnostic feedback. Future work aims to reduce computational costs for real-time use, expand to other programming languages, and integrate Large Language Models to provide actionable feedback and suggested patches.

Summary updated

Abstract

Pseudocode is extensively used in introductory programming courses to instruct computer science students in algorithm design, utilizing natural language to define algorithmic behaviors. This learning approach enables students to convert pseudocode into source code and execute it to verify their algorithms' correctness. This process typically introduces two types of errors: syntax errors and logic errors. Syntax errors are often accompanied by compiler feedback, which helps students identify incorrect lines. In contrast, logic errors are more challenging because they do not trigger compiler errors and lack immediate diagnostic feedback, making them harder to detect and correct. To address this challenge, we developed a system designed to localize logic errors within student programming assignments at the line level. Our approach utilizes pseudocode as a scaffold to build a code-pseudocode graph, connecting symbols from the source code to their pseudocode counterparts. We then employ a graph neural network to both localize and suggest corrections for logic errors. Additionally, we have devised a method to efficiently gather logic-error-prone programs during the syntax error correction process and compile these into a dataset that includes single and multiple line logic errors, complete with indices of the erroneous lines. Our experimental results are promising, demonstrating a localization accuracy of 99.2% for logic errors within the top-10 suspected lines, highlighting the effectiveness of our approach in enhancing students' coding proficiency and error correction skills.
BibTeX
@misc{xu2024logic,
  author = {Zhenyu Xu and Kun Zhang and Victor S. Sheng},
  title = {{Logic Error Localization in Student Programming Assignments Using Pseudocode and Graph Neural Networks}},
  year = {2024},
  eprint = {2410.21282},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.21282},
}
Figure