A Systematic Literature Review of Code Hallucinations in LLMs: Characterization, Mitigation Methods, Challenges, and Future Directions for Reliable AI

Abstract

This systematic literature review (SLR) investigates "code hallucinations" in Large Language Models (LLMs)—instances where models generate syntactically plausible but factually incorrect, non-existent, or non-executable code. By analyzing 60 recent papers, we categorize the root causes of these errors, highlight code-specific mitigation strategies such as execution-based feedback, and outline future directions for building more reliable AI coding assistants.

Introduction

While LLMs have shown incredible prowess in code generation, completion, and reasoning, they suffer from a critical flaw: hallucinations. In Natural Language Processing (NLP), a hallucination might involve a fabricated historical fact. However, in software engineering, code is governed by strict syntax, relies on external libraries, and must compile and execute. A code hallucination—such as an LLM inventing a non-existent API function—can lead to broken builds, logic bugs, or security vulnerabilities. This paper bridges the gap between NLP and software engineering by systematically reviewing the current landscape to define what code hallucinations are, why they happen, and how to mitigate them.

Methodology

The authors conducted a Systematic Literature Review (SLR) by querying major academic databases (ACM Digital Library, IEEE Xplore, Web of Science) and filtering results to 60 highly relevant papers. The analysis was structured around four Research Questions (RQs): defining a taxonomy of code hallucinations, reviewing standard NLP mitigations, analyzing code-specific mitigations, and cataloging evaluation benchmarks.

Expand

Source, p. 5

Taxonomy of Code Hallucinations

The review divides code hallucinations into three distinct categories:

• Factuality (Knowledge): Inventing APIs, misusing libraries, or misunderstanding language features.
• Faithfulness (Alignment): Generating code that runs but violates functional or non-functional (e.g., performance) requirements.
• Compatibility (Environment): Code that is logically correct but fails due to missing dependencies, incorrect resource paths, or version conflicts.

The authors distinguish between a simple mistake (e.g., a typo) and a hallucination. A hallucination is a fluent, highly confident fabrication of logic or APIs not grounded in reality.

Mitigation Strategies

Mitigation strategies are categorized into general NLP approaches and code-specific techniques. While standard NLP fixes like Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) are useful, code's deterministic nature allows for more robust structural enforcement.

Expand

Source, p. 3

Execution as a Superpower

The most effective mitigations use the environment as a feedback mechanism. Agent-based systems following a "Think → Code → Execute → Reflect" loop can catch hallucinations by reading compiler errors or test failures and performing self-refinement. Other successful code-specific mitigations include:

• Knowledge-Augmented Generation: RAG tailored to project-specific APIs.
• Constrained Decoding: Forcing the LLM to output valid Abstract Syntax Trees (ASTs).
• Post-Editing: Using static analysis or unit tests to verify code after generation.

Evaluation Benchmarks

The review catalogs several crucial benchmarks developed to evaluate code hallucinations:

• CodeMirage: 1,137 Python problems with hallucinated snippets and ground-truth solutions.
• CODEGUARD+: Evaluates secure and correct code generation in C++ and Python.
• HALLUCODE & CodeHaluEval: Specifically designed to measure an LLM's ability to recognize and avoid hallucinations in Python.
• CoderEval: Evaluates functional correctness using real-world projects.

The field is moving toward hallucination-specific metrics like MiHN (Micro Hallucination Number) and BuildSuccessRate.

Limitations and Future Directions

Current research is often limited by a narrow focus on Python and Java, leaving low-resource languages and proprietary enterprise codebases underrepresented. Promising future directions include:

• Explainable Detection: Tracing hallucinations back to specific prompt tokens or training data.
• Multimodal UI Code Generation: Grounding code generation to visual UI constraints.
• Unified Pipelines: Combining retrieval, structure-aware decoding, and post-generation execution verification into a single, robust pipeline.