Compilation error repair: for the student programs, from the student programs

Umair Z. Ahmed, Pawan Kumar, Amey Karkare, Purushottam Kar, Sumit GulwaniICSE (SEET) 2018 · DOI

TL;DR

This paper introduces TRACER, a novel system that automatically repairs compilation errors in student C programs. Unlike previous methods that focus on simply making code compile, TRACER aims to provide repairs that closely match what students themselves would do, making it a valuable pedagogical tool.

What Problem Does This Paper Address?

Learning to program is often hindered by cryptic compiler error messages, especially for beginners. These messages, designed for experienced developers, can be confusing and time-consuming to decipher, leading to frustration and a steep learning curve. While automated code repair exists, it often focuses on logical errors or produces generic fixes for compilation errors that might not align with a student's learning process or intended solution. This gap is particularly pronounced in large-scale educational settings like MOOCs, where personalized human assistance is infeasible.

Methods

TRACER employs a modular, three-phase approach:

  1. Error Localization: Instead of relying solely on compiler-reported line numbers, TRACER identifies the error location by considering the compiler-reported line and its immediate neighbors (one line above and one line below). This heuristic proves surprisingly effective.
  2. Abstract Code Repair: This phase uses a sequence-to-sequence prediction model (Recurrent Neural Networks with attention and LSTM) to generate an abstracted representation of the repair. Crucially, TRACER abstracts away specific literals and identifiers, replacing them with tokens representing their types (e.g., INT, FLOAT). This reduces vocabulary size and makes the system more generalizable.
  3. Concretization: The abstract repair is then translated back into concrete, compilable code. This involves mapping the abstract tokens back to actual code using information from the original source and a symbol table.

Datasets

The authors collected a dataset of over 23,000 single-line error program pairs from student submissions in an introductory C programming course at IIT-Kanpur. This dataset, derived from actual student work, differs from some prior work that artificially introduces errors into correct programs. They also used a dataset from a previous study (DeepFix) for comparison.

Key Insights and Ideas

  • Pedagogical Value of "Student-like" Repairs: The core insight is that for educational purposes, a repair that mimics a student's likely fix is more valuable than one that simply compiles. This helps students understand why their code was wrong.
  • Modular Approach Benefits: Separating error localization from repair prediction allows TRACER to use simpler, effective techniques for localization and more sophisticated deep learning models for repair, avoiding the complexity of monolithic approaches.
  • Abstraction is Key for Generalization: Abstracting code to types rather than specific tokens makes the system less sensitive to variations in variable names and literals, enabling it to handle a wider range of student code and generalize to new assignments.
  • Simple Error Localization Works: The finding that errors are often located very close to the compiler's reported line is a significant simplification for the error localization step.

Key Results

  • TRACER achieves an exact match with the student's intended repair (in abstract form) in 59.6% of cases (Prec@1).
  • When considering the top 3 recommendations, TRACER provides the correct abstract repair 66.6% of the time (Prec@3) and a relevant repair 73.85% of the time (SPrec@3).
  • For single-line errors, TRACER produces compilable code in 79.27% of cases.
  • On multi-line errors, TRACER achieves a 43.67% compilation success rate, significantly outperforming DeepFix's 27% on the same dataset.
  • The system is fast, taking an average of 1.66 seconds per repair during testing.

Limitations and Future Work

  • Global Context: TRACER currently struggles with errors requiring global context, such as missing braces or incorrect function parameter counts when the call site is distant from the function definition. Future work aims to incorporate global context.
  • Evaluation Metrics: The current metrics penalize minor syntactic variations (e.g., a != b vs. !(a == b)), even if semantically equivalent. The authors plan to develop more nuanced metrics.
  • Semantic Repair: Combining TRACER with semantic repair tools is suggested to further refine fixes.
  • Portability: While the core logic is language-agnostic, language-specific components would need development for other programming languages.

Summary updated

BibTeX
@inproceedings{ahmed2018compilation,
  author = {Umair Z. Ahmed and Pawan Kumar and Amey Karkare and Purushottam Kar and Sumit Gulwani},
  title = {{Compilation error repair: for the student programs, from the student programs}},
  booktitle = {ICSE (SEET)},
  year = {2018},
  doi = {10.1145/3183377.3183383},
}
Figure