SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization

Yue Huang, Xiangqi Wang, Xiangliang ZhangarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) often face a fundamental tension between being helpful and being trustworthy (harmless and honest). This "alignment tax" is particularly problematic in high-stakes domains like medicine or law. We present Self-Priority Alignment (SPA), a fully unsupervised framework that employs lexicographic optimization to prioritize trustworthiness over helpfulness. Through a multi-stage process involving diverse sampling, dual-criterion denoising, and uncertainty-guided preference learning, SPA enables models to align themselves without human-annotated data. Empirical evaluations on Llama-3.1 and Mistral-7B show that SPA achieves superior safety and utility across various benchmarks.

Introduction

In high-stakes scenarios, the traditional approach of balancing safety and helpfulness using weighted sums often fails. Fixed weights do not adapt to query-specific risks, potentially leading to "safety erosion" where a model compromises its integrity to be more helpful. Furthermore, high-quality human data for these trade-offs is scarce and expensive to produce.

The authors propose Priority Alignment, a paradigm shift that treats alignment as a lexicographic optimization problem. The core principle is simple: a model should be as helpful as possible, but only after it has guaranteed a baseline of safety and honesty. This ensures that trustworthiness is never traded away for utility.

Methodology: The SPA Framework

The Self-Priority Alignment (SPA) framework consists of a four-stage unsupervised pipeline designed to refine model behavior without external labels.

The four-stage Self-Priority Alignment (SPA) pipeline, including sampling, denoising, and preference construction.
  1. Diverse Sampling & Self-Refinement: The model generates multiple candidate responses using high-temperature sampling. It then self-evaluates these candidates for trustworthiness ($G_a$) and helpfulness ($G_b$), eventually producing a "refined" response aimed at maximizing both metrics.
  2. Dual-Criterion Denoising: To ensure data quality, SPA filters samples based on consistency (the refined response must outperform initial candidates) and informativeness (using score covariance to remove unstable samples).
  3. Lexicographic Preference Construction: A preference dataset is built where response $A$ is preferred over $B$ if $A$ is safer, or if they are equally safe but $A$ is more helpful.
  4. Uncertainty-Guided SimPO: The model is fine-tuned using a modified Simple Preference Optimization (SimPO) with an uncertainty-weighted loss. This forces the model to learn more decisively from training pairs with significant score gaps.

Experimental Results

The researchers evaluated SPA using Llama-3.1-8B-Instruct and Mistral-7B-Instruct across several benchmarks, including SafeRLHF, WildGuard, and HoneSet.

Win rates of SPA-aligned models compared to vanilla baselines across Harmlessness and Helpfulness metrics.

Key findings include:

  • Superior Balance: On the SafeRLHF benchmark, SPA significantly outperformed vanilla models in both harmlessness and helpfulness. For instance, SPA achieved a harmlessness score of 9.90 compared to the baseline's 9.62.
  • Generalization to New Threats: On the unseen JailbreakTrigger dataset, SPA maintained high safety (9.80) while nearly doubling the helpfulness of the baseline (6.35 vs 4.99).
  • Eliminating the Alignment Tax: General reasoning capabilities (measured by MMLU) remained stable, proving that strict safety prioritization does not degrade general intelligence.
  • High Win Rates: Head-to-head comparisons judged by GPT-4o showed that SPA-aligned models won significantly more often, particularly in honesty-critical tasks like those in the HoneSet benchmark (86% win rate on helpfulness).

Conclusion and Future Work

SPA demonstrates that LLMs can effectively "bootstrap" their own alignment. By framing the problem as a hierarchy rather than a balance, the framework provides a more interpretable and reliable way to enforce safety constraints.

However, the method's success is tied to the base model's self-evaluation capabilities. If a model cannot distinguish between safe and unsafe outputs, the self-supervision signal degrades. Future research will explore extending this lexicographic approach to other domains, such as prioritizing factual accuracy over creative expression in technical writing.

Summary updated

Abstract

In high-stakes scenarios-such as self-harm, legal, or medical queries-LLMs must be both trustworthy and helpful. However, these goals often conflict. We propose priority alignment, a new alignment paradigm that enforces a strict "trustworthy-before-helpful" ordering: optimization of helpfulness is conditioned on first meeting trustworthy thresholds (e.g., harmlessness or honesty). To realize this, we introduce Self-Priority Alignment (SPA)-a fully unsupervised framework that generates diverse responses, self-evaluates them and refines them by the model itself, and applies dual-criterion denoising to remove inconsistency and control variance. From this, SPA constructs lexicographically ordered preference pairs and fine-tunes the model using an uncertainty-weighted alignment loss that emphasizes high-confidence, high-gap decisions. Experiments across multiple benchmarks show that SPA improves helpfulness without compromising safety, outperforming strong baselines while preserving general capabilities. Our results demonstrate that SPA provides a scalable and interpretable alignment strategy for critical LLM applications.
BibTeX
@misc{huang2025spa,
  author = {Yue Huang and Xiangqi Wang and Xiangliang Zhang},
  title = {{SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization}},
  year = {2025},
  eprint = {2511.06222},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.06222},
}
Figure