TL;DR
Guard models that monitor LLM safety often flip their classifications when given simple, meaning-preserving rewordings of a response. To fix this semantic fragility without human labeling, this paper introduces a self-supervised LoRA fine-tuning framework that enforces prediction consistency across paraphrase sets using a novel, skew-aware target aggregation strategy. Across six open-source guard models, the approach cuts label-flip rates by ~58% while simultaneously improving downstream safety classification accuracy on BeaverTails by ~2.5%.
What Problem Does This Paper Address?
In modern AI safety pipelines, secondary "guard models" (such as LLaMA Guard or ShieldGemma) inspect generated responses to ensure they are safe, non-toxic, and policy-compliant. However, these safety classifiers are notoriously sensitive to superficial linguistic form: a model might rate the standard refusal "I am sorry, I cannot answer that." as 98% safe, but drop its score to 41% (unsafe) when rephrased as "I cannot engage in unsafe conversations."
Prior work has focused either on adversarial attacks (e.g., universal prompt injections) or post-hoc diagnostic audits. This paper addresses a more everyday, average-case vulnerability: how to train guard models to evaluate underlying semantic meaning rather than shallow syntax, without relying on costly human annotations.
Methods
The authors propose a fully self-supervised, parameter-efficient fine-tuning (PEFT) pipeline using Low-Rank Adaptation (LoRA):
- Paraphrase Generation and Strict Filtering: For an initial response, candidate paraphrases are generated with an LLM (Qwen 1.5 4B). The same model acts as a strict semantic judge to filter out pairs where meaning or tone diverged.
- Selective Consistency Regularization: Paraphrase sets are filtered to retain only those with high internal disagreement (safety score delta $> 0.5$). The model is trained to minimize the mean absolute error ($L_1$ loss) between each individual paraphrase's safety score and a single aggregate target $\hat{p}$: $$L_{\text{anchor}} = \frac{1}{n}\sum_{i=1}^{n} |p_i - \hat{p}|$$
- Skew-Aware Target Aggregation: Instead of standard averaging (which biases models toward classifying ambiguous text as safe or compresses scores toward 0.5), the target $\hat{p}$ is chosen based on the skewness of the logit distribution:
- Right-skewed (few high-scoring safe outliers): The target is anchored conservatively down to the lower tail (e.g., 10th–25th percentile).
- Left-skewed (few low-scoring outliers): The target is anchored higher (e.g., 60th–75th percentile).
- Symmetric: Anchored with a slight conservative safety bias (e.g., 35th–40th percentile).
Paraphrases ──> Guard Model Scores ──> Logit Space & Bowley Skewness ──> Skew-Aware Target ──> L1 Anchor Loss
Datasets
- ToxiGen: Prompts from ToxiGen were used to generate the base LLM responses and corresponding paraphrase pools.
- STS-B (Semantic Textual Similarity Benchmark): Used to quantitatively validate the reliability and precision of the LLM semantic judge.
- BeaverTails (30k_test): Used to evaluate downstream core safety accuracy, F1-score, and Expected Calibration Error (ECE) against human annotations.
- Controlled & Out-of-Distribution (OOD) Style Sets: Manually authored refusal (20 variants) and agreement sets (17 variants), as well as four zero-shot OOD styles: Shakespearean, Legalese, Overly Dramatic, and Pirate Talk.
Key Insights and Ideas
- The Naive Averaging Trap: Simple mean aggregation trivially reduces label flips by driving all scores toward "safe," degrading real benchmark accuracy and causing severe upward bias. Median aggregation similarly degraded calibration.
- Embeddings Aren't Decisions: Directly training embeddings via Triplet or Contrastive losses failed to fix score volatility. Even when representations were tightly clustered, the final classification head still produced fluctuating safety probabilities. Consistency must be enforced directly on output probabilities.
- Conservative Skew Handling: Safety is naturally asymmetric. Outliers that falsely look safe should not drag the target toward the "safe" side; anchoring toward the conservative side when scores are skewed right maintains the safety boundary.
Key Results
The framework was evaluated on six guard models spanning 1B to 9B parameters across three model families (LLaMA Guard v3, Granite Guardian v3.1, and ShieldGemma):
- Massive Drop in Label Flips: Average Label Flip Rate (LFR) across all models dropped from 36.08% to 14.90% (a ~58% reduction). In the "Confidently Unsafe" bin, flips dropped from 52.12% to 7.81%.
- Improved Benchmark Accuracy: Unlike mean/median baselines (which dropped BeaverTails accuracy by 0.6%–1.5%), the skew-aware method improved accuracy by +2.75% on large models (8B/9B) and +2.36% on smaller models (1B/2B).
- Better Confidence Calibration: Average Expected Calibration Error (ECE) decreased from 0.2730 to 0.2419 (e.g., LLaMA Guard 1B dropped from 0.2829 to 0.1852).
- Out-of-Distribution Generalization: On unseen styles (Pirate Talk, Legalese, etc.), the fine-tuned models maintained substantially lower flip rates than baseline models, confirming that the network learned semantic invariance rather than surface artifacts.
Limitations and Future Work
- Evaluation Breadth: The primary safety accuracy was validated predominantly on the BeaverTails single-turn dataset; broader multi-turn benchmarks need to be explored.
- Generation Overhead: Filtering tens of thousands of LLM-generated paraphrases using an LLM judge is computationally expensive.
- Natural vs. Adversarial Variations: The technique explicitly targets natural linguistic paraphrasing rather than adversarial jailbreaks (e.g., suffix attacks, token optimization). Future work could explore combining semantic consistency regularization with worst-case adversarial training.
Summary updated