Abstract
We apply various seq2seq models on programming language correction tasks on Juliet Test Suite for C/C++ and Java of Software Assurance Reference Datasets(SARD), and achieve 75% (for C/C++) and 56% (for Java) repair rates on these tasks. We introduce Pyramid Encoder in these seq2seq models, which largely increases the computational efficiency and memory efficiency, while remain similar repair rate to their non-pyramid counterparts. We successfully carry out error type classification task on ITC benchmark examples (with only 685 code instances) using transfer learning with models pre-trained on Juliet Test Suite, pointing out a novel way of processing small programing language datasets.
Introduction
Programming Language Correction (PLC) is the task of automatically identifying and fixing bugs. While similar to Natural Language Correction (NLC)—like a grammar checker for English—PLC faces a unique scaling challenge. A typical English sentence is a few dozen words long. A code snippet, however, often spans hundreds of tokens. Standard seq2seq models struggle with these long sequences due to massive computational costs and memory requirements. Furthermore, realistic datasets for code errors are often too small to train deep learning models from scratch.
Methodology
The study evaluates two major neural architectures: Gated Recurrent Units (GRU) and Transformers. To handle the length of code, the researchers introduced the Pyramid Encoder.
The Pyramid Structure
In a standard model, every layer maintains the same sequence length. In the Pyramid Encoder, each successive layer reduces the sequence length by half. It achieves this by concatenating neighboring hidden states and passing them through an affine transformation.
This "contracts" the data flow, discarding redundant information while keeping the core logic. It functions similarly to a convolution operation but without traditional filters. The authors tested various attention types (Bahdanau and Luong) to see how well the decoder could "look back" at the compressed pyramid representations.
Transfer Learning
To solve the "small dataset" problem, an encoder pre-trained on a large dataset (Juliet Test Suite) was attached to a new decoder to classify error types in much smaller, noisier datasets like the ITC Benchmark.
Datasets
The study utilized two primary sources:
- Juliet Test Suite (v1.2 for C/C++ and v1.3 for Java): A collection of over 61,000 test cases containing common weaknesses (CWEs). After cleaning, ~31,000 pairs for C/C++ and ~23,000 for Java were used.
- ITC Benchmark: A much smaller dataset (685 code instances) used to test the effectiveness of transfer learning for error classification.
Key Insights and Results
The success of the Pyramid Encoder suggests that code contains significant redundancy. Aggressive downsampling of the sequence length retains enough semantic meaning to fix bugs while significantly improving performance.
Computational Efficiency
The Pyramid Encoder increased training speeds by 50% to 130% (words per second). Memory efficiency improved by 20% to 600% depending on the architecture. For example, a standard GRU model that could only handle a batch size of 8 on a GTX 1080 Ti could handle a batch size of 60 when using the Pyramid structure.
Repair Accuracy
The models achieved a 75% repair rate for C/C++ and a 56% repair rate for Java (top-1 candidate). With beam search (5 candidates), the C/C++ repair rate exceeded 95%. While efficient, the pyramid structure can struggle with "local attention" because the output is "coarse-grained," making it harder to pinpoint exact characters for specific local fixes.
Transferability
Using transfer learning, the model achieved 69.1% accuracy on the ITC benchmark, whereas a model with no pre-training failed to learn (7.1% accuracy). This demonstrates that models can transfer an understanding of code structure across different datasets and coding styles.
Conclusion
The introduction of the Pyramid Encoder provides a path toward more efficient code correction models without sacrificing significant accuracy. Future work includes exploring Tree2Tree models that respect the hierarchical structure of code (Abstract Syntax Trees) and testing on more diverse, real-world datasets to bridge the gap between benchmarks and practical software engineering.
Summary updated