Localizing Malicious Outputs from CodeLLM

Mayukh Borana, Junyi Liang, Sai Sathiesh Rajan, Sudipta ChattopadhyayEMNLP 2025

TL;DR

This paper introduces FREQRANK, a novel mutation-based defense to pinpoint malicious code snippets within LLM outputs and identify their corresponding backdoor triggers. By analyzing common substrings across mutated inputs, FREQRANK effectively localizes these threats, outperforming existing methods.

What Problem Does This Paper Address?

Code Large Language Models (Code LLMs) are revolutionizing software engineering, but they are also vulnerable to sophisticated attacks like backdoors. These backdoors allow malicious actors to create models that behave normally on benign inputs but produce harmful or incorrect outputs when a specific "trigger" is present in the input. The challenge for developers is not just detecting that a model might be compromised, but precisely identifying what in the output is malicious and what in the input caused it. Existing defenses often focus on detecting the presence of a backdoor rather than localizing the specific malicious components, leaving a critical gap in understanding and mitigating these threats.

Methods

FREQRANK employs a two-pronged approach based on mutation testing and frequency analysis.

  1. Malicious Output Localization: Given an input, FREQRANK generates multiple "mutants" by slightly altering the input. These mutants are then fed to the Code LLM, and the resulting outputs are collected. The core idea is that malicious strings induced by a backdoor trigger tend to persist across these mutated outputs. FREQRANK then identifies common substrings across these outputs and ranks them based on length and frequency. The most frequent and longest common substrings are flagged as potentially malicious.
  2. Backdoor Trigger Localization: Once a malicious substring in the output is identified, FREQRANK uses the same ranking mechanism to find the corresponding backdoor trigger in the input. It identifies inputs that produced outputs containing the flagged malicious substring. These inputs are then analyzed using the substring ranking algorithm to pinpoint the common, suspicious patterns that act as triggers.

This unified ranking approach for both output localization and trigger identification is a key novelty.

Datasets

The primary dataset used for training and evaluating the models is CodeSearchNet. The authors fine-tuned their models on this dataset for three downstream tasks: code completion (CC), code generation (CG), and code summarization (CS). They also tested FREQRANK on a third-party poisoned model.

Key Insights and Ideas

  • Persistence of Malicious Strings: The fundamental assumption that malicious substrings remain consistent across mutated outputs is a powerful insight for localization.
  • Unified Localization: Using the same ranking mechanism for both output substrings and input triggers simplifies the defense and makes it more robust.
  • Robustness to False Positives: The frequency-based ranking helps FREQRANK overcome false positives (benign inputs accidentally triggering malicious outputs) by prioritizing patterns that appear consistently across multiple inputs.
  • Scalability with Mutants: The effectiveness of FREQRANK increases significantly as the number of generated mutants increases, suggesting a trade-off between computational cost and detection accuracy.
  • Adaptability to Complex Triggers: FREQRANK shows promise in handling multi-trigger backdoor attacks, although sample efficiency might decrease with more triggers.

Key Results

  • High Attack Success Rate (ASR): The poisoned models achieved an average ASR of 86.6% across three downstream tasks and three base models (CodeLlama, CodeGemma, Gemini 2.5 Flash). Gemini 2.5 Flash showed a particularly high ASR of 98.2% with a low false positive rate (FPR) of 0.2%.
  • Effective Localization: FREQRANK successfully identified malicious outputs within the top five suggestions in 98% of cases. The malicious substring was ranked in the first position nearly 70% of the time.
  • Trigger Localization Efficiency: FREQRANK could isolate backdoor triggers with as few as four inputs, even with a 50% false positive rate.
  • Improved Performance: FREQRANK was 35-50% more effective than baseline methods like RAP and length-based sorting in detecting backdoors.
  • Impact on Model Performance: Backdooring led to a decrease in BLEU4 scores on clean inputs, averaging 13.7% across models, indicating some performance degradation.

Limitations and Future Work

  • Generalizability to Non-Substring Backdoors: FREQRANK might struggle with backdoor attacks that don't rely on inserting specific substrings into the output (e.g., deleting lines).
  • Sample Efficiency: While robust, the sample efficiency of FREQRANK can decrease with an increasing number of triggers or a higher rate of false positives.
  • Computational Cost: Generating multiple mutants for each input adds computational overhead.
  • Future Work: The authors hope FREQRANK will inspire further research into defending against backdoors in LLMs and have made their code and data publicly available. They suggest exploring its application to other downstream tasks and poisoning regimes.

Summary updated

BibTeX
@inproceedings{borana2025localizing,
  author = {Mayukh Borana and Junyi Liang and Sai Sathiesh Rajan and Sudipta Chattopadhyay},
  title = {{Localizing Malicious Outputs from CodeLLM}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure