An interpretable error correction method for enhancing code-to-code translation

Min Xue, Artur Andrzejak, Marla LeutherICLR 2024 · OpenReview

TL;DR

Researchers have introduced $k$NN-ECD, a post-hoc error correction method that boosts code-to-code translation accuracy without the need for expensive model retraining. By combining a $k$-nearest neighbor search over a specialized datastore with a "unified name rule" to focus on code logic, they improved Java-to-Python translation accuracy from 68.9% to a staggering 89.9%.

What Problem Does This Paper Address?

Transformer-based models have revolutionized program translation (e.g., converting Java to Python), but they face two major hurdles:

  1. The "Black Box" Problem: These models are notoriously difficult to interpret. When a translation fails, it’s nearly impossible to trace which part of the training data led to the error.
  2. Diminishing Returns on Retraining: Improving these models usually requires massive computational resources for retraining, often resulting in only marginal gains.

Furthermore, source code is full of "rare identifiers"—unique variable or function names that appear only once. These act as noise, distracting models from the underlying logic and structure of the code.

Methods

The authors propose a two-pronged approach to "repair" the output of existing models like TransCoder-ST:

Overview of the kNN-ECD and kNN-ECSm architecture
  • $k$NN-ECD (Error Correction Datastore): Instead of retraining the Transformer, the authors build an external datastore of "error-correction pairs." When the model produces a translation, the system uses a $k$-nearest neighbor search to find similar snippets in the datastore. If a match is found, it "overwrites" the model's wrong tokens with corrected ones.
  • $k$NN-ECS$_m$: To enhance this further, they developed a distributed system where $m$ smaller datastores are connected in series. If the first "expert" datastore fails to fix the code (verified by unit tests), the code is passed to the next one for another round of correction.
  • Unified Name Rule: To solve the noise problem, they implement a preprocessing step that replaces specific identifiers (like user_age_count) with generic placeholders (like b0, b1). This forces the $k$NN search to focus on the logic and structure of the code rather than being distracted by unique naming conventions.

Datasets

The researchers used a robust pipeline to build their error-correction knowledge:

  • Source: Java source code from Google BigQuery.
  • Test Generation: They used EvoSuite to generate high-quality unit tests for the Java functions.
  • The "Gold Standard": By running the original Java code through TransCoder-ST and checking the Python output against the unit tests, they identified 26,230 instances where the model failed initially but could be corrected. This formed the basis of their error-correction datastore.

Key Insights and Ideas

  • Logic Trumps Names: The "Unified Name Rule" is a simple but powerful insight. By "anonymizing" code, the $k$NN search becomes much more effective at finding structurally similar solutions. It turns out that for error correction, how a function works is more important than what the variables are called.
  • Interpretability by Retrieval: Because the system retrieves specific snippets to justify a correction, it provides a "decision-making basis." A developer can see exactly which piece of code from the datastore influenced the fix, making the system much more transparent than a standard Transformer.
  • Serial Experts: The serial structure of $k$NN-ECS$_m$ mimics a multi-round peer review. By breaking the datastore into smaller chunks and iterating, the system captures a more comprehensive range of repair information that a single, monolithic datastore might miss.

Key Results

The quantitative improvements are significant:

Performance comparison showing the accuracy jump to 89.9%
  • Accuracy Jump: The baseline TransCoder-ST model had a translation accuracy of 68.9%. Applying the $k$NN-ECD/ECS$_m$ methods pushed this to 89.9% for Java-to-Python translation.
  • Outperforming Baselines: The method significantly outperformed other common tools like j2py (38.3%) and specialized models like DOBF (52.7%).
  • Efficiency: These gains were achieved without any fine-tuning or retraining of the underlying large language model, representing a high-accuracy, low-cost alternative to traditional model scaling.

Limitations and Future Work

While the results are impressive, the authors note a few areas for growth:

  • Oracle Dependency: The current system relies on an "oracle" (unit tests) to determine if a translation has failed and needs correction. In real-world scenarios where unit tests aren't available, the system would need a different trigger for the correction phase.
  • Retrieval Latency: Searching through massive datastores can be slow. While they suggest clustering to improve speed, high-latency retrieval remains a challenge for real-time applications.
  • Broadening the Scope: The study focused primarily on Java-to-Python. Future work could explore how well this "neighbor-based" correction scales across more diverse or obscure programming languages.

Summary updated

Abstract

Transformer-based machine translation models currently dominate the field of model-based program translation. However, these models fail to provide interpretative support for the generated program translations. Moreover, researchers frequently invest substantial time and computational resources in retraining models, yet the improvement in translation accuracy is quite limited. To address these issues, we introduce a novel approach, $k\text{NN-ECD}$, which combines $k$-nearest-neighbor search with a key-value error correction datastore to overwrite the wrong translations of TransCoder-ST. This provides a decision-making basis for interpreting the corrected translations. Building upon this, we further propose $k\text{NN-ECS}_{m}$, a methodology that employs a distributed structure with $m$ sub-datastores connected in series, utilizing $m$ diverse experts for multi-round error correction. Additionally, we put forward a unified name rule, encouraging the datastore to focus more on code logic and structure rather than diverse rare identifiers. Our experimental results show that our approach improves the translation accuracy from 68.9\% to 89.9\% of TransCoder-ST (for translation from Java to Python). This error correction method augments program translation, overcoming the inherent limitations of Transformer-based code translation models, such as resource-intensive retraining requirements and uninterpretable outcomes.
BibTeX
@inproceedings{xue2024interpretable,
  author = {Min Xue and Artur Andrzejak and Marla Leuther},
  title = {{An interpretable error correction method for enhancing code-to-code translation}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=fVxIEHGnVT},
}
Figure