Abstract
Large Language Models (LLMs) often face a fundamental tension between being helpful and being trustworthy (harmless and honest). This "alignment tax" is particularly problematic in high-stakes domains like medicine or law. We present Self-Priority Alignment (SPA), a fully unsupervised framework that employs lexicographic optimization to prioritize trustworthiness over helpfulness. Through a multi-stage process involving diverse sampling, dual-criterion denoising, and uncertainty-guided preference learning, SPA enables models to align themselves without human-annotated data. Empirical evaluations on Llama-3.1 and Mistral-7B show that SPA achieves superior safety and utility across various benchmarks.
Introduction
In high-stakes scenarios, the traditional approach of balancing safety and helpfulness using weighted sums often fails. Fixed weights do not adapt to query-specific risks, potentially leading to "safety erosion" where a model compromises its integrity to be more helpful. Furthermore, high-quality human data for these trade-offs is scarce and expensive to produce.
The authors propose Priority Alignment, a paradigm shift that treats alignment as a lexicographic optimization problem. The core principle is simple: a model should be as helpful as possible, but only after it has guaranteed a baseline of safety and honesty. This ensures that trustworthiness is never traded away for utility.
Methodology: The SPA Framework
The Self-Priority Alignment (SPA) framework consists of a four-stage unsupervised pipeline designed to refine model behavior without external labels.
- Diverse Sampling & Self-Refinement: The model generates multiple candidate responses using high-temperature sampling. It then self-evaluates these candidates for trustworthiness ($G_a$) and helpfulness ($G_b$), eventually producing a "refined" response aimed at maximizing both metrics.
- Dual-Criterion Denoising: To ensure data quality, SPA filters samples based on consistency (the refined response must outperform initial candidates) and informativeness (using score covariance to remove unstable samples).
- Lexicographic Preference Construction: A preference dataset is built where response $A$ is preferred over $B$ if $A$ is safer, or if they are equally safe but $A$ is more helpful.
- Uncertainty-Guided SimPO: The model is fine-tuned using a modified Simple Preference Optimization (SimPO) with an uncertainty-weighted loss. This forces the model to learn more decisively from training pairs with significant score gaps.
Experimental Results
The researchers evaluated SPA using Llama-3.1-8B-Instruct and Mistral-7B-Instruct across several benchmarks, including SafeRLHF, WildGuard, and HoneSet.
Key findings include:
- Superior Balance: On the SafeRLHF benchmark, SPA significantly outperformed vanilla models in both harmlessness and helpfulness. For instance, SPA achieved a harmlessness score of 9.90 compared to the baseline's 9.62.
- Generalization to New Threats: On the unseen JailbreakTrigger dataset, SPA maintained high safety (9.80) while nearly doubling the helpfulness of the baseline (6.35 vs 4.99).
- Eliminating the Alignment Tax: General reasoning capabilities (measured by MMLU) remained stable, proving that strict safety prioritization does not degrade general intelligence.
- High Win Rates: Head-to-head comparisons judged by GPT-4o showed that SPA-aligned models won significantly more often, particularly in honesty-critical tasks like those in the HoneSet benchmark (86% win rate on helpfulness).
Conclusion and Future Work
SPA demonstrates that LLMs can effectively "bootstrap" their own alignment. By framing the problem as a hierarchy rather than a balance, the framework provides a more interpretable and reliable way to enforce safety constraints.
However, the method's success is tied to the base model's self-evaluation capabilities. If a model cannot distinguish between safe and unsafe outputs, the self-supervision signal degrades. Future research will explore extending this lexicographic approach to other domains, such as prioritizing factual accuracy over creative expression in technical writing.
Summary updated