Conditioned Language Policy: A General Framework for Steerable Multi-Objective Finetuning

Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, Alekh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Avinava Dubey, Alexandre Ramé, Johan Ferret, Geoffrey Cideron, Le Hou, Hongkun Yu, Amr Ahmed, Aranyak Mehta, Léonard Hussenot, Olivier Bachem, Edouard Leurent

Abstract

Large Language Models (LLMs) often face conflicting objectives, such as safety versus helpfulness or conciseness versus detail. Traditional finetuning methods typically optimize for a fixed trade-off, requiring costly retraining to adjust model behavior. We present Conditioned Language Policy (CLP), a framework that enables a single model to be steered across a continuum of objectives at inference time. By employing parameter-space conditioning and multi-task training, CLP achieves superior Pareto performance and steerability compared to existing prompt-based and weight-interpolation methods.

Introduction

Aligning LLMs with human preferences is inherently a multi-objective problem. Because human preferences are diverse and context-dependent, a static model optimized for a single "average" preference often fails to satisfy individual user needs. Current approaches to multi-objective finetuning (MOFT) either rely on prompts—which often suffer from "mode collapse"—or weight interpolation (e.g., Rewarded Soups), which can be sub-optimal as the constituent models are not trained for joint performance.

The CLP Framework

Conditioned Language Policy (CLP) addresses these limitations by integrating two core components: parameter-space conditioning and multi-task training.

Conceptual diagram of the Multi-Objective Language Model and its Pareto performance compared to baselines.

Parameter-Space Conditioning

Instead of relying solely on textual prompts to guide behavior, CLP maintains sets of conditioned parameters. At inference time, these parameters are linearly blended based on a user-defined weight vector $w$, representing the desired balance between objectives, and a KL-regularization weight $\alpha$. This allows the model's internal representations to adapt directly to the specified trade-off.

Multi-Task Training

To ensure the model can navigate the entire trade-off space (the Pareto front), CLP is trained on a distribution of reward weightings. During training, weights are sampled from a continuum, forcing the model to learn optimal policies for various objective combinations. This bridges the gap found in zero-shot interpolation methods, where models trained on individual objectives may not overlap in action space.

Experimental Results

The researchers evaluated CLP on text summarization tasks using XSum and TLDR datasets, scaling from T5-Small (60M parameters) to T5-XL (3B).

Pareto Dominance and Steerability

CLP variants consistently outperformed baselines, demonstrating "Pareto dominance." For any given level of one metric (e.g., factuality via NLI), CLP provided higher performance on the secondary metric (e.g., quality via ROUGE) than prompt-based or interpolation methods. Furthermore, CLP models exhibited significantly greater steerability, covering a wider range of the reward space without collapsing into a single behavioral mode.

Efficiency of Attention Conditioning

A key finding is the effectiveness of attn-CLP, which conditions only the attention layers. As shown in the reward-KL curves, attn-CLP achieves performance nearly identical to full-CLP (which conditions all parameters) while being significantly more parameter-efficient.

Reward-KL curves comparing different CLP variants, highlighting the efficiency of attn-CLP.

Win Rates

Automated evaluation using Gemini 1.0 Ultra confirmed these results. attn-CLP and full-CLP showed an 11-16% improvement in win rates over multi-task prompting and a 5-10% improvement over Rewarded Soups, highlighting the practical benefits of the framework.

Conclusion

Conditioned Language Policy provides a robust and flexible framework for multi-objective LLM alignment. By moving conditioning into the parameter space and utilizing multi-task training, CLP enables the creation of truly steerable models that can adapt to diverse human preferences on the fly. Future work will explore non-linear reward combinations and instance-dependent adaptive algorithms.

Summary updated

BibTeX
@inproceedings{wang2024conditioned,
  author = {Kaiwen Wang and Rahul Kidambi and Ryan Sullivan and Alekh Agarwal and Christoph Dann and Andrea Michi and Marco Gelmi and Yunxuan Li and Raghav Gupta and Avinava Dubey and Alexandre Ramé and Johan Ferret and Geoffrey Cideron and Le Hou and Hongkun Yu and Amr Ahmed and Aranyak Mehta and Léonard Hussenot and Olivier Bachem and Edouard Leurent},
  title = {{Conditioned Language Policy: A General Framework for Steerable Multi-Objective Finetuning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.118},
}
Figure