Application of Seq2Seq Models on Code Correction

Shan Huang, Xiao Zhou, Sang ChinFrontiers Artif. Intell 2021 · OpenReview

TL;DR

This paper introduces a "pyramid encoder" for Seq2Seq models to efficiently correct logic errors in code. This novel architecture significantly speeds up training and reduces memory usage without sacrificing accuracy, achieving impressive repair rates on C/C++ and Java code. Furthermore, they demonstrate the effectiveness of transfer learning for code error classification on small datasets.

What Problem Does This Paper Address?

The paper tackles the challenging problem of automatic code correction, specifically focusing on logic errors in programs that have already passed compilation. While syntax errors are relatively easy to detect and fix, logic errors (like memory leaks, redundant code, or incorrect variable usage) are much harder to identify and correct. These errors can be subtle, require a deep understanding of the code's execution, and their "error space" is vast, with tools like Error-Prone identifying hundreds of bug patterns. Traditional rule-based methods for fixing these errors are labor-intensive and often case-specific. Machine learning approaches offer a promising alternative, but a significant hurdle is the scarcity of large, realistic datasets for training.

Methods

The core of this work lies in applying Seq2Seq models (encoder-decoder architectures) to code correction. These models, successful in natural language processing tasks like translation, are adapted here to translate buggy code into corrected code.

The key innovation is the "pyramid encoder." In standard Seq2Seq models, each layer's output is directly fed to the next. The pyramid encoder, however, introduces a module that effectively halves the input sequence length at each encoding layer. This is achieved by processing pairs of neighboring hidden states. This "coarse-graining" of information significantly reduces computational and memory costs, especially for long code sequences, without a substantial drop in performance.

The authors also explore transfer learning as a strategy to overcome the data scarcity problem. They leverage a model pre-trained on a larger dataset (Juliet Test Suite) and use its encoder as a feature extractor for a new, smaller dataset (ITC benchmark) for a different task (error type classification).

Datasets

  • Juliet Test Suite for C/C++ (v1.2): This is the primary dataset used for training and evaluating the code correction models. It contains over 61,000 test cases with flawed and repaired code, covering over 100 Common Weakness Enumerations (CWEs). The authors performed preprocessing, including removing dead code and unrealistic "if conditions," resulting in 31,082 good-bad code instance pairs.
  • Juliet Test Suite for Java (v1.3): Used to test the generality of their models, this dataset yielded 23,015 preprocessed instance pairs.
  • ITC benchmark: A smaller dataset (566 C/C++ code instances) used for the error type classification task. It contains 44 error categories.

Key Insights and Ideas

  • Seq2Seq models are viable for code correction: The paper demonstrates that Seq2Seq architectures, proven in NLP, can be effectively applied to the complex task of fixing logic errors in code.
  • The "Pyramid Encoder" is a game-changer for efficiency: The core contribution is the pyramid encoder, which dramatically reduces computational and memory requirements for Seq2Seq models when processing long sequences like code. This makes training on large code datasets more feasible.
  • Code is different from natural language: The paper highlights that code instances are typically much longer than natural language sentences, making efficient processing techniques like the pyramid encoder particularly valuable.
  • Transfer learning is crucial for small datasets: The success in error type classification on the small ITC benchmark underscores the power of transfer learning, enabling effective model training even with limited data.
  • Rule-based vs. ML-based PLC: The authors draw an analogy between rule-based PLC and statistical machine translation, suggesting that ML approaches offer advantages in self-sufficiency and potential for self-improvement.

Key Results

  • Repair Rates:
    • C/C++ Juliet Test Suite: Achieved 75% repair rate with a single candidate and higher with multiple candidates.
  • Java Juliet Test Suite: Achieved 56% repair rate.
  • Pyramid Encoder Efficiency:
    • Training Speed: Increased by 50%-100% compared to regular encoders.
  • Memory Efficiency: Increased by up to 600%.
  • Error Type Classification (ITC benchmark):
    • Transfer learning with a pre-trained encoder achieved ~60-69% accuracy.
  • Models trained from scratch on the ITC dataset showed no predictive power.
  • Robustness to Length: Seq2Seq models (with or without pyramid encoder) showed robustness to longer input lengths, with performance drops primarily attributed to data scarcity.

Limitations and Future Work

  • Dataset Limitations: The authors acknowledge that publicly available datasets are often small and noisy, which is a general challenge in the field. They also note that the Juliet Test Suite is generated and may not perfectly reflect real-world bugs.
  • Pyramid Encoder and Local Attention: The pyramid encoder did not perform well with Luong's local attention mechanism, likely due to the "coarse-grained" nature of the encoder output blurring local information. This attention mechanism was excluded from further discussion.
  • Future Work: The authors suggest exploring different architectures (e.g., parallel encoders/decoders, Tree2Seq models), applying their models to other tasks like program generation and optimization, and investigating the differences between artificial and realistic datasets. They also plan to further investigate the influence of various architectures.

Summary updated

Abstract

We apply various seq2seq models on programming language correction tasks on Juliet Test Suite for C/C++ and Java of Software Assurance Reference Datasets and achieve 75% (for C/C++) and 56% (for Java) repair rates on these tasks. We introduce pyramid encoder in these seq2seq models, which significantly increases the computational efficiency and memory efficiency, while achieving similar repair rate to their nonpyramid counterparts. We successfully carry out error type classification task on ITC benchmark examples (with only 685 code instances) using transfer learning with models pretrained on Juliet Test Suite, pointing out a novel way of processing small programming language datasets.
BibTeX
@inproceedings{huang2021application,
  author = {Shan Huang and Xiao Zhou and Sang Chin},
  title = {{Application of Seq2Seq Models on Code Correction}},
  booktitle = {Frontiers Artif. Intell},
  year = {2021},
  url = {https://openreview.net/forum?id=r5gcKtIYvm},
}
Figure