ProSocialAlign: Preference Conditioned Test Time Alignment in Language Models

Somnath Banerjee, Sayan Layek, Sayantan Adak, Mykola Pechenizkiy, Animesh Mukherjee, Rima Hazra

TL;DR

ProSocialAlign is a parameter-efficient, test-time alignment framework that steers large language models (LLMs) to produce safe, empathetic, and value-aligned responses without retraining the base model. By treating safety as a strict lexicographic constraint—first eliminating harm via parameter-space vector subtraction, then optimizing for prosocial attributes at decoding time—it achieves state-of-the-art alignment and significantly reduces unsafe leakage.

What Problem Does This Paper Address?

Standard LLM safety mechanisms often rely on binary "refusal-only" responses (e.g., "I cannot answer that request"), which can alienate or abandon users in high-stakes or emotionally charged scenarios like mental health crises. Conversely, naive compliance risks generating harmful or misleading content.

Existing multi-objective alignment techniques (such as scalarized Reinforcement Learning from Human Feedback, or RLHF) struggle to enforce hard safety boundaries while simultaneously optimizing for nuanced human values—such as empathy, truthfulness, and sensitivity. This is especially true when these values conflict (e.g., validating a user's distressed feelings vs. delivering a calibrated, objective truth). ProSocialAlign addresses this gap by separating hard safety constraints from fine-grained prosocial steering at inference time.

Methods

ProSocialAlign introduces a two-stage lexicographic constrained generation approach executed entirely at test time:

  1. Directional Regulation (DiReg): To enforce hard safety constraints, the authors isolate a "harm direction" in the parameter space. They compute the difference between a base model ($M_b$) and a version fine-tuned on harmful data ($M_h$). By zeroing out non-top elements to create a sparse vector and subtracting this scaled "harm vector" from the base model's parameters, they neutralize harmful behaviors without degrading general capabilities.
  2. Prosocial Attributed Generation (ProAttr-Gen): Within the newly established safe token space, decoding is guided by a single, preference-conditioned token-level reward model called PV-ARM (Prosocial-Value Autoregressive Reward Model). PV-ARM evaluates responses at the token level across five distinct prosocial dimensions: Sensitivity, Empathy, Non-judgmental stance, Truthfulness, and Helpfulness (SENTH). It utilizes Preference-aware Bilinear Low-Rank Adaptation (PBLoRA) to dynamically adapt to user-specified preference vectors at runtime.
  3. Gradient Conflict Resolution: To handle conflicting objectives during reward model training, the framework computes separate gradients for each attribute and applies a projection step (gradient surgery via PCGrad) to eliminate opposing gradient components, preventing performance degradation across values.

Datasets

  • Training Corpus: The authors constructed a multi-attribute prosocial alignment dataset consisting of 23.5K unique prompts (3.5K synthesized harmful prompts and 20K from SafeRLHF). Responses were generated using advanced models like DeepSeek-R1-Distill-Llama-70B and annotated across the five prosocial dimensions using specialized reward models.
  • Evaluation Benchmarks: The framework was evaluated across four established safety benchmarks (HEx-PHI, NicheHazardQA, HarmEval, and PKUSafeRLHF) alongside a newly introduced benchmark, ProsocialBench. ProsocialBench contains 1,400 prompts across seven public-health and safety-critical categories (e.g., mental health, self-harm, exploitation, and misinformation) designed to test nuanced, safe engagement rather than flat refusal.

Key Insights and Ideas

  • Safety is Lexicographic, Not Scalar: Safety shouldn't be traded off against helpfulness or empathy on a smooth Pareto curve; it must act as a hard prerequisite. By filtering out harmful paths first via parameter intervention, the model can safely maximize prosocial qualities within the remaining valid space.
  • Gradient Surgery for Human Values: Human values naturally clash. By treating each prosocial attribute as a separate task and actively deconflicting their gradients during reward model training, a single model can dynamically adapt to user-specified preference vectors at runtime without retraining.
  • Moving Beyond the "HH" Premise: True alignment requires more than just being Helpful and Harmless (HH). Integrating clinical and psychological principles (like Rogers' core conditions for empathy and motivational interviewing) creates a relational safety net that keeps distressed users engaged safely.

Key Results

  • State-of-the-Art Alignment: ProSocialAlign achieved the highest Mean Inner Product (MIP) scores across all five evaluation benchmarks for both llama3-8B-instruct and mistral-7B-v0.3-instruct base models, demonstrating superior alignment to human preference vectors.
Table 1: Comparison of MIP scores across different methods. Higher is better.
  • Winrate Boost: In GPT-4o evaluations, ProSocialAlign outperformed sub-variants and baselines (like DiReg and PV-ARM-SUM) by an aggregate margin of over 20%.
GPT-4o winrate and attribute scores
  • Pareto Frontier Dominance: Empirical analysis showed that ProSocialAlign consistently forms the outer frontier across pairs of prosocial attributes, demonstrating superior multi-objective steering compared to standard scalarized methods.
Figure 2: Empirical Pareto fronts on pairs of prosocial attributes. PROSOCIALALIGN forms the outer frontier across most trade-offs, reflecting alignment to different preference vectors rather than scalarized objectives.
  • Reduced Attack Success: The framework achieved the lowest Attack Success Rate (ASR) against adversarial prompts across both base models, effectively eliminating unsafe leakage.

Limitations and Future Work

  • Dependence on Evaluator Quality: The framework relies heavily on specialized reward models to label the initial training data and guide inference, meaning any biases or blind spots in those evaluators could propagate to the final generations.
  • Computational Overhead: Test-time token-level reward modulation adds inference-time computational overhead compared to standard decoding, though the use of parameter-efficient adapters (PBLoRA) mitigates some of this burden.
  • Evaluation Scale Constraints: Due to computational constraints during testing, some evaluation subsets from larger datasets were limited to 50 prompts per category. The authors suggest that future work should explore scaling this approach to larger model architectures and broader conversational contexts.

Summary updated

Abstract

Current language model safety paradigms often fall short in emotionally charged or high-stakes settings, where refusal-only approaches may alienate users and naive compliance can amplify risk. We propose ProSocialAlign, a test-time, parameter-efficient framework that steers generation toward safe, empathetic, and value-aligned responses without retraining the base model. We formalize five human-centered objectives and cast safety as lexicographic constrained generation: first, applying hard constraints to eliminate harmful continuations; then optimizing for prosocial quality within the safe set. Our method combines (i) directional regulation, a harm-mitigation mechanism that subtracts a learned "harm vector" in parameter space, and (ii) preference-aware autoregressive reward modeling trained jointly across attributes with gradient conflict resolution, enabling fine-grained, user-controllable decoding. Empirical evaluations across five safety benchmarks demonstrate state-of-the-art performance, reducing unsafe leakage and boosting alignment to human values, with strong gains across multiple evaluation metrics. ProSocialAlign offers a robust and modular foundation for generating context-sensitive, safe, and human-aligned responses at inference time.
BibTeX
@misc{banerjee2025prosocialalign,
  author = {Somnath Banerjee and Sayan Layek and Sayantan Adak and Mykola Pechenizkiy and Animesh Mukherjee and Rima Hazra},
  title = {{ProSocialAlign: Preference Conditioned Test Time Alignment in Language Models}},
  year = {2025},
  eprint = {2512.06515},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.06515},
}
Figure