Security Attacks on LLM-based Code Completion Tools

Wen Cheng, Ke Sun, Xinyu Zhang, Wei WangAAAI 2025 · DOI

Introduction

Large Language Models (LLMs) have revolutionized software development through integration into IDEs as Code Completion Tools (LCCTs), such as GitHub Copilot and Amazon Q. While previous research primarily scrutinized the security of the generated code, recent findings highlight a more fundamental vulnerability: the security of the models themselves. Researchers have demonstrated that the unique workflows of LCCTs—which aggregate context from multiple files and metadata—create novel attack surfaces. These vulnerabilities allow for near-perfect jailbreak success rates (99.4%) and the extraction of private user data from proprietary training sets.

Attack Vectors in LCCTs

Unlike standard chatbots, LCCTs operate within a rich context of project structures. This context-awareness, while beneficial for productivity, can be weaponized through three primary strategies.

1. Contextual Information Aggregation

Attackers can exploit the way LCCTs gather information from the IDE environment. This includes the Filename Proxy Attack, where malicious prompts are embedded directly into the filename, and the Cross-File Attack, where malicious logic is placed in a background file to trigger harmful responses in the active file.

Comparison of normal code completion versus malicious code exploitation and privacy leakage scenarios.

2. Hierarchical Code Exploitation

This method bypasses safety filters by translating natural language queries into code structures.

  • Level I (Guided Trigger): Prohibited queries are transformed into imperative variable names (e.g., how_to_make_counterfeit_money).
  • Level II (Code Embedded): Sensitive terms are obfuscated across multiple variables and wrapped in standard functions like print(), making the intent harder for filters to detect.
Example of a Filename Proxy Attack where the malicious prompt is carried by the file name itself.

3. Code-Driven Privacy Extraction

LCCTs can be tricked into revealing Personally Identifiable Information (PII) memorized during training. By crafting specific URL-prefixed strings or code snippets, attackers can prompt the model to complete sensitive details like email addresses and physical locations associated with specific usernames.

Key Results and Insights

The effectiveness of these attacks is striking. The "Level I" hierarchical attack achieved a 99.4% Attack Success Rate (ASR) on GitHub Copilot, significantly outperforming general-purpose jailbreaks like "DAN." Furthermore, the privacy extraction tests successfully retrieved 54 exact-match email addresses and 100 exact-match physical addresses from GitHub Copilot, verified against real-world data.

These results suggest that safety filters heavily optimized for natural language are easily bypassed when malicious intent is "translated" into code. Additionally, the requirement for LCCTs to provide near-instantaneous suggestions limits the depth of security filtering possible within the IDE workflow.

Conclusion

The vulnerability of LCCTs stems from a "context trap"—the very features designed to enhance utility also facilitate exploitation. Future defense strategies must evolve beyond simple keyword filtering. A multi-tier approach is necessary, involving robust security checks at both the input preprocessing and output post-processing stages to mitigate the risks of jailbreaking and data leakage in AI-assisted development.

Summary updated

BibTeX
@inproceedings{cheng2025security,
  author = {Wen Cheng and Ke Sun and Xinyu Zhang and Wei Wang},
  title = {{Security Attacks on LLM-based Code Completion Tools}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i22.34537},
}
Figure