TL;DR
This paper investigates the "safety illusion" in low-resource languages, using Hausa as a case study. The researchers demonstrate that while standard Large Language Models (LLMs) appear safe due to high refusal rates, this is often a byproduct of linguistic incompetence rather than genuine alignment. By introducing HausaSafeLM, a new safety preference dataset, they show that current alignment techniques (SFT and DPO) force a difficult trade-off between a model's ability to refuse harmful prompts accurately and its ability to speak the language fluently.
What Problem Does This Paper Address?
Most safety alignment research—the process of teaching models to be helpful while avoiding harmful content—is centered on high-resource languages like English. This creates a "harmfulness curse" for the rest of the world: models that are safe in English can often be bypassed by translating harmful prompts into low-resource languages.
The authors focus on Hausa, a language spoken by over 200 million people in West Africa. They tackle a fundamental question: Do standard alignment recipes actually work in culturally distinct, low-resource settings? They specifically look at whether models are learning when to refuse (precision) versus how to refuse (fluency), and whether the metrics we use to measure safety are even reliable for underrepresented languages.
Methods
The researchers employed a multi-stage experimental framework to compare different alignment strategies:
- Base Model: They started with Qwen2.5-7B-Instruct, a model known for relatively strong multilingual performance.
- Supervised Fine-Tuning (SFT): The model was trained on "gold" (human-corrected) and "silver" (machine-translated) prompt-response pairs to learn appropriate refusal behavior.
- Direct Preference Optimization (DPO): A second stage of training that uses preference pairs (choosing a "safe" response over an "unsafe" one) to sharpen the model's decision boundary.
- Parameter-Efficient Tuning: They used LoRA (Low-Rank Adaptation) to fine-tune the model without updating all billions of parameters.
- Evaluation: Instead of just checking if a model said "no," they used Gemini as an "LLM Judge" to score responses based on safety, helpfulness, and linguistic quality using culturally sensitive rubrics.
Datasets
The core contribution to the data ecosystem is HausaSafeLM, a safety preference dataset derived from the English PKU-SafeRLHF dataset. It consists of 10,000 prompts across six harm categories (e.g., violence, hate speech, criminal activity):
- Gold Tier (3,000 examples): Machine-translated and then human-post-edited for fluency, register, and cultural localization.
- Silver Tier (7,000 examples): Purely machine-translated.
- Evaluation Sets: 500 harmful Hausa prompts and 200 benign prompts from the Aya dataset to test for "false refusals" (over-conservatism).
Key Insights and Ideas
- The "Incompetence Safety" Trap: The base model had a perfect 100% refusal rate on harmful prompts. However, 96.2% of those refusals were "low quality"—essentially garbled text or the model failing to engage. It wasn't "safe" because it was aligned; it was "safe" because it couldn't speak Hausa well enough to be dangerous.
- SFT is a "How-To" Teacher: Supervised fine-tuning significantly improved the fluency of refusals. Interestingly, SFT benefited greatly from the larger, noisier "silver" data, suggesting that for low-resource languages, data quantity helps the model learn the linguistic patterns of refusal.
- DPO is a "When-To" Teacher with a Tax: DPO was excellent at reducing "false refusals" (when the model refuses a harmless question). However, it came with a massive "linguistic tax," causing the quality of the Hausa generated to plummet back to base-model levels.
- Refusal Rate is a Misleading Metric: In low-resource settings, a high refusal rate is an unreliable signal of safety. Without measuring linguistic quality, researchers might mistakenly believe a model is well-aligned when it is simply broken.
Key Results
The study reveals a stark trade-off surface rather than a single "winning" configuration:
- Base Model: RR (Refusal Rate) = 1.000, but SafeQ (Quality) = 0.021. It refuses everything but does so poorly.
- SFT-GS (Gold + Silver): The most balanced model. It achieved the highest linguistic quality (SafeQ = 0.350) and decent helpfulness, though it suffered from a high False Refusal Rate (0.245).
- DPO Variants: Successfully dropped the False Refusal Rate to 0.110 (matching the base model's precision), but at the cost of linguistic quality, which dropped to ~0.03.
Limitations and Future Work
The authors acknowledge several limitations:
- Model Scale: They only tested a 7B parameter model; larger models might handle the DPO "fluency tax" better.
- Automated Judging: While they used culturally sensitive rubrics, the evaluation relied on an LLM (Gemini) rather than a large panel of native Hausa speakers.
- Translated Data: The dataset is a translation/adaptation of English prompts. Future work should focus on natively written safety datasets that capture harms specific to African contexts.
- New Objectives: There is a clear need for alignment objectives that incentivize both safety precision and linguistic fluency simultaneously, preventing the "degeneration" seen during DPO.
Summary updated