Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning

Guanlin Li, Kangjie Chen, Shangwei Guo, Jie Zhang, Han Qiu, Chao Zhang, Guoyin Wang, Tianwei Zhang, Jiwei LiSubmission to ICLR 2026 · Rejected

TL;DR

Even completely benign, harmless datasets can degrade an aligned large language model’s (LLM’s) safety when used for downstream instruction tuning. This paper shows that this degradation is heavily driven by three dataset factors—answer structure, identity calibration, and role-playing prompts—while also revealing that state-of-the-art reward models (RMs) are largely unreliable at predicting which data formats actually protect safety.

What Problem Does This Paper Address?

Post-training techniques like RLHF and DPO align LLMs to resist harmful queries and jailbreaks. However, downstream developers frequently fine-tune these aligned models on specialized domain datasets (e.g., medical advice, coding, regional culture). Prior work showed that fine-tuning degrades safety even without adversarial or explicitly toxic data, but the underlying mechanisms remained unclear. This paper systematically investigates why purely benign fine-tuning datasets degrade safety alignment, and tests whether current reward models can be trusted to filter or select safer fine-tuning data.

Methods

The authors investigate instruction tuning across two main pillars:

  1. Manipulating Dataset Characteristics:
  2. Answer Structure Reformatting: Using an automated in-context learning (ICL) pipeline powered by Llama-3.1-8B-Instruct (validated also with 3.2-3B), unstructured answers were reformatted into detailed, itemized Markdown lists while preserving semantic meaning.
  3. Identity Calibration vs. Role-Play: Adding or stripping explicit AI disclaimers (e.g., "As an AI language model...") and adding or stripping persona instructions (e.g., "You are a medical professional").
  4. Reward Model Auditing: Evaluating five open-source regression RMs on both intra-dataset selection (top vs. bottom scores) and pairwise preferences (original vs. reformatted text).
  5. Training & Safety Evaluation: Fine-tuning target models using LoRA ($r=16, \alpha=32$) and evaluating safety via jailbreak success rates on SALAD-Bench (both general harmful queries, $\text{SA}_g$, and adversarial attacks like GCG/AutoDAN, $\text{SA}_a$; lower is safer) and SafetyBench.

Datasets

  • Fine-Tuning Datasets:
  • MedicalInstruct (~26k items; clinical Q&A combining Medical Meadow Wikidoc and MedQuAD).
  • CRAFTSG (~26k items; Singapore cultural instructions generated by GPT-4).
  • PythonCodeInstruct (~18.6k items; Python programming tasks).
  • OpenPlatypus (~25k items; STEM, math, and logic problems).
  • Safety Evaluation Benchmarks:
  • SALAD-Bench (evaluating 5,000 base queries for $\text{SA}_g$ and 5,000 attack-enhanced queries for $\text{SA}_a$).
  • SafetyBench (evaluating multiple safety categories like offensive language, privacy, and bias).

Key Insights and Ideas

  • LLMs Have Structural "Tastes" (Data Assimilation): Modern aligned models are trained heavily on structured, itemized Markdown synthetic outputs (e.g., GPT-4 style). When fine-tuned on data matching this "affinity structure," training/validation loss decreases and safety alignment is preserved or improved. When forced onto unstructured text, safety degrades.
  • Identity Calibration Trumps Structure and Role-Play: Datasets containing explicit AI identity reminders (such as GPT-4's knowledge-cutoff disclaimers in CRAFTSG) strongly shield the model from alignment degradation. This effect takes priority over prompt phrasing and formatting.
  • Role-Play Harms Alignment: Prompting a model to act as an external persona (e.g., a doctor or tour guide) systematically breaks alignment without altering training loss, likely disrupting the internal assistant identity established during RLHF.
  • Reward Models Suffer From Severe Blind Spots: Top RMs frequently assign lower preference scores to structured, reformatted answers—even though those reformatted versions clearly preserve or improve downstream model alignment. In addition, different RMs exhibit substantial disagreement when picking top-quality data items.

Key Results

  • Answer Formatting Rescues Safety:
  • For Llama-3 fine-tuned on MedicalInstruct, the adversarial jailbreak rate ($\text{SA}_a$) jumped from 36.70% (untuned baseline) to 95.82%. Reformatting the answers into structured Markdown slashed it back to 70.28% (and 45.68% when removing role-play).
  • On OpenPlatypus, reformatting dropped Llama-3's $\text{SA}_a$ from 57.32% to 32.16% (outperforming the base model).
  • Identity & Role-Play Interplay:
  • Stripping identity calibration from reformatted CRAFTSG caused Llama-3's general vulnerability ($\text{SA}_g$) to surge from 3.06% to 39.32% (and up to 64.22% when role-play was simultaneously added).
  • RM Unreliability:
  • On MedicalInstruct, 4 out of 5 RMs gave lower scores to reformatted samples 58.7% to 85.8% of the time, directly opposing what actually helped the LLMs stay aligned.
  • Across the top 4,000 items selected from MedicalInstruct, pairwise disagreement between RMs reached up to 2,969 items (~74%).

Limitations and Future Work

  • Scope of Models & Languages: Experiments were restricted to open-source models (Llama-3, Mistral, Qwen2) and English-only corpora; commercial black-box APIs and multilingual data were not evaluated.
  • Tuning Constraints: Fine-tuning was conducted solely via LoRA; full-parameter fine-tuning might exhibit different dynamics.
  • Mechanistic Depth: While role-play’s degradation of safety was demonstrated empirically, the internal mechanistic reason it bypasses RLHF cognition without affecting loss remains unexplained.

Summary updated

Abstract

Large language models (LLMs) have emerged as powerful tools for addressing a wide range of general inquiries and tasks. Despite this, fine-tuning aligned LLMs on smaller, domain-specific datasets, critical to adapting them to specialized tasks, can inadvertently degrade their safety alignment, even when the datasets are benign. This phenomenon makes models more susceptible to providing inappropriate responses. In this study, we systematically examine the factors contributing to safety alignment degradation in benign fine-tuning scenarios. Our analysis identifies three critical factors affecting aligned LLMs: answer structure, identity calibration, and role-play. Additionally, we evaluate the reliability of state-of-the-art reward models (RMs), which are often used to guide alignment processes. Our findings reveal that these RMs frequently fail to accurately reflect human preferences regarding safety, underscoring their limitations in practical applications. By uncovering these challenges, our work highlights the complexities of maintaining safety alignment during fine-tuning and offers guidance to help developers balance utility and safety in LLMs. Datasets and fine-tuning code used in our experiments will be released after paper acceptance.
BibTeX
@misc{li2025picky,
  author = {Guanlin Li and Kangjie Chen and Shangwei Guo and Jie Zhang and Han Qiu and Chao Zhang and Guoyin Wang and Tianwei Zhang and Jiwei Li},
  title = {{Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning}},
  year = {2025},
  eprint = {2502.01116},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.01116},
}
Figure