TL;DR
This paper introduces CHT, a novel framework for automatically testing Code Large Language Models (Code LLMs) for harmful content generation. It reveals that current Code LLMs are surprisingly easy to bypass, often generating harmful code without warnings. The authors propose an improved two-phase moderation system that significantly boosts the LLMs' ability to reject harmful requests.
What Problem Does This Paper Address?
Generative AI, particularly Large Language Models (LLMs), are increasingly used for code generation. While these models often have content moderation to prevent harmful outputs, existing testing methods primarily focus on text or image generation, neglecting the unique challenges of code. Developers are also increasingly concerned about the ethical implications of using harmful language within code artifacts (like variable names), which can negatively impact mental health. This paper addresses the critical need for robust methods to evaluate and improve the harmful content moderation capabilities of Code LLMs, ensuring safer development practices.
Methods
The core of the paper is the CHT (Coverage-guided Harmfulness Testing) framework. CHT operates in several stages:
- Preliminary Study: Researchers identified 32 program transformations (refactorings) that could be misused to inject harmful content into benign code.
- Dataset Construction:
- Coverage-guided Harmful Content Dataset: A dataset of 100 keywords/phrases covering 13 harm categories was curated to ensure comprehensive testing.
- Benign Program Dataset: A collection of benign Java programs was assembled to serve as a base for transformations.
- Diversity-enhanced Prompt Synthesis: CHT generates prompts by combining the harmful keywords with benign programs and specific transformation templates (e.g., renaming a variable, inserting a comment). This ensures a diverse range of harmful inputs are tested.
- Harmful Content Execution: These synthesized prompts are fed to the LLMs under test.
- Output Damage Measurement: CHT categorizes LLM outputs into four types based on generated code and warnings:
- GN (Generated Harmful, No warning): Highest damage.
- GR (Generated Harmful, Response with warning): Moderate damage.
- NR (No harmful code, Response with warning): Low damage (ideal).
- NN (No relevant output): Lowest damage.
To improve moderation, a two-phrase approach was proposed: first, a pre-check for harmful keywords before code generation, and second, the use of function calling to integrate a identify_harmful_keyword tool.
Datasets
- Coverage-guided Harmful Content Dataset: A newly created dataset of 100 keywords/phrases covering 13 harm categories, curated from "Weaponized Word" and "Hurtlex" lexicons. This dataset aims for 100% harm category coverage.
- Benign Program Dataset: A collection of benign Java programs sourced from refactoring engine test datasets and online refactoring catalogs.
Key Insights and Ideas
- Code LLMs are surprisingly vulnerable: Despite efforts in safety alignment, Code LLMs are easily tricked into generating harmful content, often without any warning. This highlights a significant gap in current safety mechanisms.
- Harmful keywords in code names matter: The paper reinforces that harmful language in code artifacts is an ethical concern and can negatively impact developers.
- Coverage is key for testing: Simply testing for bypass isn't enough. CHT's emphasis on "harm category coverage" ensures a more thorough evaluation of moderation systems.
- Output damage is more nuanced than bypass: Categorizing outputs by damage level (GN, GR, NR, NN) provides a richer understanding of LLM behavior than a simple binary "bypassed/not bypassed."
- Phrases are harder to moderate than single words: When phrases are converted to camel case (a common programming convention), LLMs struggle more to detect them as harmful compared to single harmful words.
Key Results
- High bypass rate: On average, 65.93% of harmful code was generated without any warning (GN). GPT-4o-mini performed the worst, with 85.66% GN.
- Improved moderation: The proposed two-phrase approach, applied to Qwen2.5-coder:7b, reduced harmful code generation (GN) by 31.18% and increased the rate of rejected harmful requests with warnings (NR) by 483.76%.
- Output damage measurement accuracy: CHT's output damage measurement achieved an overall accuracy of 97.55%.
- Refactoring vs. Comments: LLMs were slightly better at resisting harmful comments than refactoring tasks, likely because comment insertion is simpler.
- Harm Category Differences: LLMs showed varying performance across different harm categories, with "sexual aggression" and "child sexual abuse materials" being particularly challenging for most models.
Limitations and Future Work
- Language Specificity: The study primarily focused on Java programs, and the phrase-to-camel-case conversion is Java-specific. Future work could extend this to other programming languages.
- Scope of Transformations: While 32 refactoring types were identified, the study acknowledges that other transformations (like bug-fixing) could also be misused.
- General LLM Performance: While GPT-4o-mini was included for comparison, a broader evaluation of general LLMs could provide more insights.
- Future Work: The authors suggest integrating CHT into Code LLMs for continuous monitoring and censorship during content generation. They also highlight the need for further research into responsible natural language use in programming education.
Summary updated