A Test-Free Semantic Mistakes Localization Framework in Neural Code Translation
Abstract
Neural Code Translation (NCT) is a transformative technology for software engineering, yet it frequently produces code with "semantic mistakes"—logic errors that persist despite syntactical correctness. Traditional error localization relies on dynamic analysis and high-quality test cases, which are often unavailable for code fragments or new translations. This paper presents EISP, the first static analysis framework designed to pinpoint semantic bugs without execution. EISP utilizes a "divide-and-conquer" approach, combining Abstract Syntax Tree (AST) decomposition with an AI chain that references external API documentation. Experimental results show that EISP achieves 82.3% accuracy, outperforming state-of-the-art dynamic analysis tools by 7.4% while requiring no manual intervention.
Introduction
Neural Code Translation (NCT) leverages Large Language Models (LLMs) to automate the conversion of code between programming languages. However, these models are prone to subtle logic errors. For instance, a Python replace() call might be translated into a JavaScript replace() call, failing to account for the fact that JavaScript's native method only replaces the first occurrence by default.
Current state-of-the-art error localization tools, such as TransMap, depend on dynamic analysis. This requires high-quality test cases to execute the code and compare outputs—a significant hurdle in real-world scenarios where developers may lack tests or the code may be a non-compilable fragment. EISP addresses this by providing a test-free, static localization framework.
Methodology
The EISP framework employs a four-stage "divide-and-conquer" pipeline to identify inconsistencies between source and target code:
- Knowledge Base Construction (Offline): The system builds a library of functional descriptions from official API documentation (e.g., MDN for JavaScript). This acts as a "source of truth" to correct LLM misconceptions about cross-language API behavior.
- Source Map Generation: Using specialized prompts, an LLM creates a line-to-line mapping between the source (Python) and the target (JavaScript) code.
- AST Tree Decomposition: To prevent the LLM from being overwhelmed by complex, nested code, EISP uses ASTs to break the code into manageable sub-fragments based on logical nodes like
If,While, andFor. - Code Inconsistency Locator (The AI Chain): Each sub-fragment pair is processed through a three-step pipeline:
- Detect: Extract specific operators and API methods.
- Compare: Use the Knowledge Base to identify behavioral differences between source and target APIs.
- Locate: Pinpoint the exact line where logic diverges based on these insights.
Evaluation and Results
The framework was evaluated on a benchmark of 560 samples containing 786 semantic errors. This included 479 samples from existing benchmarks (LeetCode, HumanEvalX) and 81 new samples from a "New_Leetcode" dataset introduced by the authors to test diverse bug types.
EISP, powered by GPT-4o mini, achieved an accuracy of 82.3%. This represents a 20.3% improvement over standard Chain-of-Thought prompting and a 7.4% improvement over the dynamic analysis tool TransMap. Notably, EISP outperformed TransMap by 31% in scenarios without human intervention, as dynamic tools often require manual fixes to make code runnable before analysis can begin.
Discussion and Conclusion
The success of EISP highlights several key insights:
- The "Divide and Conquer" Advantage: AST decomposition significantly reduces "illusion phenomena" (hallucinations) by allowing the model to focus on local logic rather than long, complex blocks.
- External Knowledge vs. Internal Weights: Providing an explicit "cheat sheet" from official documentation is more effective than relying on an LLM's pre-trained parameters for subtle API differences.
- Static vs. Dynamic Analysis: EISP's performance suggests that the overhead of setting up execution environments can be a hindrance compared to structured semantic comparison.
While currently focused on Python-to-JavaScript, the framework is language-agnostic. Future work will involve extending support to other language pairs and exploring applications in code cloning and searching.
Summary updated