A test-free semantic mistakes localization framework in Neural Code Translation

Lei Chen, Sai Zhang, Fangzhou Xu, Zhenchang Xing, Liang Wan, Xiaowang Zhang, Zhiyong FengarXiv 2024 · DOI · arXiv

A Test-Free Semantic Mistakes Localization Framework in Neural Code Translation

Abstract

Neural Code Translation (NCT) is a transformative technology for software engineering, yet it frequently produces code with "semantic mistakes"—logic errors that persist despite syntactical correctness. Traditional error localization relies on dynamic analysis and high-quality test cases, which are often unavailable for code fragments or new translations. This paper presents EISP, the first static analysis framework designed to pinpoint semantic bugs without execution. EISP utilizes a "divide-and-conquer" approach, combining Abstract Syntax Tree (AST) decomposition with an AI chain that references external API documentation. Experimental results show that EISP achieves 82.3% accuracy, outperforming state-of-the-art dynamic analysis tools by 7.4% while requiring no manual intervention.

Introduction

Neural Code Translation (NCT) leverages Large Language Models (LLMs) to automate the conversion of code between programming languages. However, these models are prone to subtle logic errors. For instance, a Python replace() call might be translated into a JavaScript replace() call, failing to account for the fact that JavaScript's native method only replaces the first occurrence by default.

Example of semantic mistakes in Python-to-JavaScript translation.

Current state-of-the-art error localization tools, such as TransMap, depend on dynamic analysis. This requires high-quality test cases to execute the code and compare outputs—a significant hurdle in real-world scenarios where developers may lack tests or the code may be a non-compilable fragment. EISP addresses this by providing a test-free, static localization framework.

Methodology

The EISP framework employs a four-stage "divide-and-conquer" pipeline to identify inconsistencies between source and target code:

  1. Knowledge Base Construction (Offline): The system builds a library of functional descriptions from official API documentation (e.g., MDN for JavaScript). This acts as a "source of truth" to correct LLM misconceptions about cross-language API behavior.
  2. Source Map Generation: Using specialized prompts, an LLM creates a line-to-line mapping between the source (Python) and the target (JavaScript) code.
  3. AST Tree Decomposition: To prevent the LLM from being overwhelmed by complex, nested code, EISP uses ASTs to break the code into manageable sub-fragments based on logical nodes like If, While, and For.
  4. Code Inconsistency Locator (The AI Chain): Each sub-fragment pair is processed through a three-step pipeline:
  5. Detect: Extract specific operators and API methods.
  6. Compare: Use the Knowledge Base to identify behavioral differences between source and target APIs.
  7. Locate: Pinpoint the exact line where logic diverges based on these insights.
The EISP framework architecture.

Evaluation and Results

The framework was evaluated on a benchmark of 560 samples containing 786 semantic errors. This included 479 samples from existing benchmarks (LeetCode, HumanEvalX) and 81 new samples from a "New_Leetcode" dataset introduced by the authors to test diverse bug types.

EISP, powered by GPT-4o mini, achieved an accuracy of 82.3%. This represents a 20.3% improvement over standard Chain-of-Thought prompting and a 7.4% improvement over the dynamic analysis tool TransMap. Notably, EISP outperformed TransMap by 31% in scenarios without human intervention, as dynamic tools often require manual fixes to make code runnable before analysis can begin.

Discussion and Conclusion

The success of EISP highlights several key insights:

  • The "Divide and Conquer" Advantage: AST decomposition significantly reduces "illusion phenomena" (hallucinations) by allowing the model to focus on local logic rather than long, complex blocks.
  • External Knowledge vs. Internal Weights: Providing an explicit "cheat sheet" from official documentation is more effective than relying on an LLM's pre-trained parameters for subtle API differences.
  • Static vs. Dynamic Analysis: EISP's performance suggests that the overhead of setting up execution environments can be a hindrance compared to structured semantic comparison.

While currently focused on Python-to-JavaScript, the framework is language-agnostic. Future work will involve extending support to other language pairs and exploring applications in code cloning and searching.

Summary updated

Abstract

In the task of code translation, neural network-based models have been shown to frequently produce semantically erroneous code that deviates from the original logic of the source code. This issue persists even with advanced large models. Although a recent approach proposed using test cases to identify these semantic errors, it relies heavily on the quality of the test cases and is not applicable to code snippets without test cases in real-world scenarios. Therefore, We present EISP, a static analysis framework based on the Large Language Model (LLM).First, the framework generates a semantic mapping between source code and translated code. Next, each sub-code fragment is identified by recursively traversing the abstract syntax tree of the source code, and its corresponding translated code fragment is found through the semantic mapping. Finally, EISP connects each pair of sub-code fragments with fine-grained knowledge hints through an AI chain to assist LLMs in discovering semantic mistakes in the translated code. In our benchmark evaluation, the EISP framework, based on GPT-4o mini, achieved an accuracy of 82.3\%, representing a 20.3\% improvement over baseline methods using the same base model, and a 7.4\% improvement compared to dynamic analysis methods that require test cases and manual intervention. To our knowledge, EISP is the first tool to locate semantic errors in translated code without test cases or compilable code. This innovative tool provides the software engineering community with a new way to deal with code fragments without test cases.
BibTeX
@misc{chen2024test,
  author = {Lei Chen and Sai Zhang and Fangzhou Xu and Zhenchang Xing and Liang Wan and Xiaowang Zhang and Zhiyong Feng},
  title = {{A test-free semantic mistakes localization framework in Neural Code Translation}},
  year = {2024},
  eprint = {2410.22818},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.22818},
}
Figure