Abstract
Large Language Models (LLMs) often face conflicting objectives, such as safety versus helpfulness or conciseness versus detail. Traditional finetuning methods typically optimize for a fixed trade-off, requiring costly retraining to adjust model behavior. We present Conditioned Language Policy (CLP), a framework that enables a single model to be steered across a continuum of objectives at inference time. By employing parameter-space conditioning and multi-task training, CLP achieves superior Pareto performance and steerability compared to existing prompt-based and weight-interpolation methods.
Introduction
Aligning LLMs with human preferences is inherently a multi-objective problem. Because human preferences are diverse and context-dependent, a static model optimized for a single "average" preference often fails to satisfy individual user needs. Current approaches to multi-objective finetuning (MOFT) either rely on prompts—which often suffer from "mode collapse"—or weight interpolation (e.g., Rewarded Soups), which can be sub-optimal as the constituent models are not trained for joint performance.
The CLP Framework
Conditioned Language Policy (CLP) addresses these limitations by integrating two core components: parameter-space conditioning and multi-task training.
Parameter-Space Conditioning
Instead of relying solely on textual prompts to guide behavior, CLP maintains sets of conditioned parameters. At inference time, these parameters are linearly blended based on a user-defined weight vector $w$, representing the desired balance between objectives, and a KL-regularization weight $\alpha$. This allows the model's internal representations to adapt directly to the specified trade-off.
Multi-Task Training
To ensure the model can navigate the entire trade-off space (the Pareto front), CLP is trained on a distribution of reward weightings. During training, weights are sampled from a continuum, forcing the model to learn optimal policies for various objective combinations. This bridges the gap found in zero-shot interpolation methods, where models trained on individual objectives may not overlap in action space.
Experimental Results
The researchers evaluated CLP on text summarization tasks using XSum and TLDR datasets, scaling from T5-Small (60M parameters) to T5-XL (3B).
Pareto Dominance and Steerability
CLP variants consistently outperformed baselines, demonstrating "Pareto dominance." For any given level of one metric (e.g., factuality via NLI), CLP provided higher performance on the secondary metric (e.g., quality via ROUGE) than prompt-based or interpolation methods. Furthermore, CLP models exhibited significantly greater steerability, covering a wider range of the reward space without collapsing into a single behavioral mode.
Efficiency of Attention Conditioning
A key finding is the effectiveness of attn-CLP, which conditions only the attention layers. As shown in the reward-KL curves, attn-CLP achieves performance nearly identical to full-CLP (which conditions all parameters) while being significantly more parameter-efficient.
Win Rates
Automated evaluation using Gemini 1.0 Ultra confirmed these results. attn-CLP and full-CLP showed an 11-16% improvement in win rates over multi-task prompting and a 5-10% improvement over Rewarded Soups, highlighting the practical benefits of the framework.
Conclusion
Conditioned Language Policy provides a robust and flexible framework for multi-objective LLM alignment. By moving conditioning into the parameter space and utilizing multi-task training, CLP enables the creation of truly steerable models that can adapt to diverse human preferences on the fly. Future work will explore non-linear reward combinations and instance-dependent adaptive algorithms.
Summary updated