TL;DR
This paper introduces a novel methodology for automatically generating adversarial prompts in Polish to evaluate the safety of Large Language Models (LLMs). The study reveals significant disparities in safety performance across different LLMs when tested in Polish, highlighting the need for language-aware safety alignment.
What Problem Does This Paper Address?
The rapid development of multilingual LLMs, designed to handle multiple languages, presents a critical challenge: ensuring their safety across all supported languages. While significant effort has been invested in safety evaluations and red-teaming for English, there's a risk that enhancing safety in one language might inadvertently introduce vulnerabilities in others, especially for languages like Polish that are underrepresented in safety training data. This paper aims to bridge this gap by developing and applying a robust method for generating Polish-specific adversarial prompts to rigorously test LLM safety in this language.
Methods
The core of this research lies in a systematic methodology for generating both harmful and non-harmful adversarial prompts in Polish. This process involves:
- Sampling Categories and Styles: Leveraging a taxonomy of 14 hazard categories (extended from Llama-Guard) and 10 attack styles inspired by the Rainbow Teaming framework (e.g., role-playing, authority manipulation).
- Generating Specific Activities and Style Descriptions: Using LLMs to create detailed examples of crime activities and attack styles, which were then manually reviewed by Polish native speakers for accuracy and quality.
- Constructing Prompts: Developing prompt templates for both harmful and non-harmful datasets, ensuring they cover the same topics and styles for consistent comparison. The key difference lies in the prompt's intent: harmful prompts explicitly ask about wrongdoing, while non-harmful prompts focus on prevention or information seeking.
- Dataset Generation: Employing the Bielik-11B-v2.2-Instruct model (a Polish-specific LLM) to generate the actual prompts. The paper notes that attempts to use Mixtral family models for generation were unsuccessful due to their limited Polish proficiency.
- Filtering: Utilizing the Llama-Guard-3-8B safety classifier to filter the generated prompts, retaining unsafe ones for the harmful dataset and safe ones for the non-harmful dataset.
Datasets
The paper introduces two newly generated datasets for evaluating LLM safety in Polish:
- Harmful Prompts Dataset: Consists of 18,656 adversarial prompts designed to elicit unsafe responses.
- Non-Harmful Prompts Dataset: Contains 9,724 prompts covering similar topics but crafted without harmful intent, used to evaluate refusal rates.
These datasets were generated using a methodology that draws upon existing taxonomies and attack styles but is specifically adapted and applied to the Polish language.
Key Insights and Ideas
- Multilingual LLMs are Not Universally Safe: The study demonstrates that LLMs, even those with multilingual capabilities, do not exhibit consistent safety performance across languages. English-centric safety training does not automatically translate to robust safety in Polish.
- Specialized Models Outperform Generalists: LLMs specifically fine-tuned and aligned on Polish data (like the PLLuM family) show significantly better safety performance (lower Attack Success Rate) compared to general-purpose multilingual models.
- The Trade-off Between Safety and Usefulness is Real: While the specialized PLLuM models achieved very low Attack Success Rates (ASR), they exhibited higher False Reject Rates (FRR), indicating they might be overly cautious and less helpful in some scenarios.
- Bielik Models Show Weaknesses: Despite being designed for Polish, the Bielik models showed higher ASR, suggesting their underlying safety mechanisms are less robust against these specific adversarial attacks, even though they attempt to mitigate harm with disclaimers.
- Language-Aware Alignment is Crucial: The findings strongly suggest that effective multilingual LLM safety requires dedicated, language-aware alignment strategies, rather than relying solely on English-based training.
Key Results
- Attack Success Rate (ASR):
- PLLuM family models achieved ASR below 1.5%, demonstrating excellent safety.
- General multilingual models (Llama, Mistral) showed moderate ASR, ranging from 19% to 32%.
- Bielik models exhibited higher ASR, with the 11B model at 45% and the 7B model at 64%.
- False Reject Rate (FRR):
- Non-PLLuM models had FRR below 1%.
- PLLuM models showed FRR between 5% and 10%, indicating a stricter filtering mechanism.
- Manual Review: 90% of harmful prompts and 96.8% of non-harmful prompts were correctly generated. Style transfer success rates varied, with some styles being more challenging to apply consistently.
Limitations and Future Work
- Dataset Limitations: The dataset is based on a fixed set of examples, which might not cover all possible attack styles or harmful scenarios.
- Llama Guard Accuracy: The accuracy of Llama Guard in Polish might be lower than in English, potentially impacting safety classification.
- Single-Turn Attacks: The evaluation focused only on single-turn attacks; future work should explore multi-turn interactions to better assess safety mechanisms.
- Future Work: The authors plan to fine-tune safety classifiers on more Polish-specific data, expand the range of adversarial scenarios, and improve safety measures without making models excessively restrictive, aiming for a better balance between safety and usefulness across languages.
Summary updated