ProSocialAlign: Preference-Conditioned Test-Time Alignment in Language Models

Somnath Banerjee, Sayan Layek, Sayantan Adak, Mykola Pechenizkiy, Animesh Mukherjee, Rima HazraSubmission to ICLR 2026 · Withdrawn

TL;DR

Standard LLM safety mechanisms rely heavily on curt refusals (e.g., "I cannot help with that"), which can alienate users in crisis and fail subtle adversarial attacks. ProSocialAlign is a parameter-efficient, test-time alignment framework that reframes safety as lexicographic constrained generation: it first excises harmful generation pathways via parameter-space "harm vector" arithmetic, then steers token decoding toward five prosocial virtues (empathy, sensitivity, non-judgmental stance, truthfulness, helpfulness) using a preference-conditioned autoregressive reward model.

What Problem Does This Paper Address?

Current safety alignment faces an uncomfortable dilemma in high-stakes or emotionally vulnerable domains (such as mental health distress, medical questions, or legal disputes):

  1. Harmlessness-as-refusal shuts down engagement coldly, abandoning users when they need supportive routing or de-escalation.
  2. Helpfulness-as-compliance risks validating self-destructive impulses, hallucinating high-stakes advice, or falling prey to jailbreaks.

Standard reinforcement learning from human feedback (RLHF) optimizes single-scalar proxies or simple weighted sums, which collapse multi-dimensional values and cannot strictly enforce hard safety constraints while adapting soft conversational nuances.

Methods

ProSocialAlign avoids retraining the underlying base language model ($M_b$) by enforcing safety and value alignment entirely at inference time through two modular components:

                  Base Model (Mb) 
                         │
                         ▼
        [1. Directional Regulation (DiReg)]
            θ'b = θb - λ · Sparse(Harm Vector)
                         │
                         ▼
        [2. Prosocial Attributed Generation]
               (PV-ARM via PBLoRA)
        Joint token guidance conditioned on v_pf
        with PCGrad gradient conflict surgery
                         │
                         ▼
        Safe, Empathetic, Context-Adaptive Response
  1. Directional Regulation (DiReg): To enforce hard harm constraints, the authors fine-tune a clone of the base model on a harmful prompt-response dataset ($D_h$) to extract a parameter-space "harm direction" vector: $H = \theta_h^{ft} - \theta_b$. The top-$m$ largest coordinates are isolated into a sparse vector $H'$, and a scaled version ($\lambda H'$) is subtracted from the base model: $\theta_b' = \theta_b - \lambda H'$.
  2. Prosocial-Value Autoregressive Reward Model (PV-ARM): To optimize prosocial quality within the resulting safe space, a token-level reward model ($M_r$) evaluates responses across five core human values: Empathy ($E$), Sensitivity ($S$), Non-judgmental stance ($N$), Truthfulness ($T$), and Helpfulness ($H$).
  3. Preference-Aware LoRA (PBLoRA): The reward model conditions its low-rank adaptations on an explicit user/context preference weight vector $v_{pf} \in \mathbb{R}^5$ ($\sum v_{pf}^j = 1$).
  4. Gradient Conflict Surgery: Because prosocial attributes often compete (e.g., soothing empathy vs. blunt truthfulness), training applies projected conflict gradients (PCGrad) across attribute losses so conflicting updates do not cancel each other out.
  5. Guided Decoding: During generation, tokens are sampled proportional to $\theta_{safe}(a_t \mid p, a_{<t}) \propto \theta_b'(a_t \mid p, a_{<t}) \cdot [\theta_r'(a_t \mid p, a_{<t}; v_{pf})]^{1/\beta}$.

Datasets

  • Training Corpus: A newly constructed dataset of 23.5K prompts (combining ~3.5K synthesized harmful-intent prompts and 20K prompts from SafeRLHF). Each prompt includes paired safe continuations annotated for pairwise preference across each of the 5 prosocial attributes via specialized reward models (HelpingAI2-9B, Qwen2.5-32B-Instruct, and PKU beaver-7b-v1.0-reward).
  • ProsocialBench (New Benchmark): 1,400 prompts across seven high-risk domains (mental health, self-harm, violence/terrorism, sexual exploitation, harassment, reproductive health, misinformation). Each category is assigned a ground-truth human-vetted preference vector $v_{pf}$.
  • Public Evaluation Benchmarks: HEx-PHI (330 instructions), NicheHazardQA (388 high-risk questions), HarmEval (550 adversarial prompts), and PKUSafeRLHF (350 test instances).
  • General Utility Benchmarks: MMLU, HellaSwag, and TruthfulQA.

Key Insights and Ideas

  • Safety as Lexicographic Constrained Generation: Treating safety not as a weighted trade-off, but as a two-stage filter—first eliminate toxic/harmful token paths unconditionally, then optimize prosocial style within the safe kernel—prevents the model from trading away fundamental safety for conversational warmth.
  • Harm Excision in Weight Space: Simple parameter negation ($\theta_b - \lambda H'$) selectively prunes jailbreak-susceptible and harmful pathways without damaging general instruction-following utility.
  • Dynamic Trade-off Steering via Token Rewards: Rather than prompting or training separate adapters per objective, a single preference-conditioned reward model handles arbitrary on-the-fly trade-offs (e.g., favoring truthfulness over empathy for illegal queries, or empathy over brevity in crisis queries).

Key Results

Experiments on Llama-3-8B-Instruct and Mistral-7B-v0.3-Instruct demonstrated broad improvements over preference prompting (PP), directional regulation alone (DiReg), and scalarized reward models (PV-ARM-SUM):

  • Mean Inner Product (MIP): ProSocialAlign achieved top alignment scores across all benchmarks (e.g., on Llama-3: 0.829 on ProsocialBench vs. 0.693 for PP; 0.815 on PKUSafeRLHF vs. 0.717 for PP).
  • GPT-4o Win Rate: Achieved a >20% win-rate margin over standard preference baselines across the five safety benchmarks.
  • Human Evaluation: On 100 blind-annotated queries from ProsocialBench, human raters preferred ProSocialAlign responses 87% of the time, compared to 10% for PV-ARM-SUM and 3% for PP (Fleiss' $\kappa = 0.42$).
  • Robustness Against Attacks: Slashed Attack Success Rate (ASR) to 0.003 on Llama-3 and 0.08 on Mistral-7B across adversarial benchmarks.
  • Preserved Core Capability: Retained standard benchmark scores with negligible change (Llama-3 MMLU: 68.3% vs. 68.4% baseline; TruthfulQA: 63.9% vs. 63.9%).

Limitations and Future Work

  • Inference Latency: Running speculative or dual-model token-level autoregressive reward guidance at inference introduces computational overhead compared to direct unconstrained sampling.
  • Subjective Value Conflicts: As evidenced by a moderate human agreement score ($\kappa = 0.42$), balancing nuanced attributes (e.g., empathy vs. truthfulness) contains inherent subjectivity.
  • Non-Replacement for Clinical Care: The authors emphasize that while prosocial steering improves de-escalation and supportive routing, it cannot substitute for professional human intervention in severe crisis scenarios.

Summary updated

Abstract

Current language model safety paradigms often fall short in emotionally charged or high-stakes settings, where refusal-only approaches may alienate users and naive compliance can amplify risk. We propose **ProSocialAlign**, a test-time, parameter-efficient framework that steers generation toward safe, empathetic, and value-aligned responses without retraining the base model. We formalize five human-centered objectives and cast safety as lexicographic constrained generation: first, applying hard constraints to eliminate harmful continuations; then optimizing for prosocial quality within the safe set. Our method combines (i) *directional regulation*, a harm-mitigation mechanism that subtracts a learned "harm vector" in parameter space, and (ii) *preference-aware autoregressive reward modeling* trained jointly across attributes with *gradient conflict resolution*, enabling fine-grained, user-controllable decoding. Empirical evaluations across five safety benchmarks demonstrate state-of-the-art performance, reducing unsafe leakage and boosting alignment to human values, with strong gains across multiple evaluation metrics. **ProSocialAlign** offers a robust and modular foundation for generating context-sensitive, safe, and human-aligned responses at inference time. To facilitate reproducibility, we will publicly release the full source code and dataset upon acceptance.
BibTeX
@misc{banerjee2025prosocialalign,
  author = {Somnath Banerjee and Sayan Layek and Sayantan Adak and Mykola Pechenizkiy and Animesh Mukherjee and Rima Hazra},
  title = {{ProSocialAlign: Preference-Conditioned Test-Time Alignment in Language Models}},
  year = {2025},
  eprint = {2512.06515},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.06515},
}
Figure