In standard reinforcement learning (RL) and AI alignment, the typical approach to optimization is to collapse all objectives into a single scalar reward. Whether balancing speed against safety in autonomous driving, or helpfulness against harmlessness in large language models (LLMs), engineers traditionally assign fixed weights to each sub-objective and sum them up.
However, as argued by Peter Vamplew et al. in Scalar reward is not enough: a response to Silver, Singh, Precup and Sutton (2021), forcing multi-dimensional human values into a single scalar hard-wires a specific trade-off into the agent's weights. If user preferences shift, or if different stakeholders require different balances, the entire model must be retrained from scratch.
To solve this, a powerful paradigm has emerged: preference-conditioned policies. Instead of training a model to maximize a fixed trade-off, researchers train a single universal policy conditioned directly on a dynamic preference vector. This allows users to navigate the entire Pareto-optimal frontier in real time using a single model instance.
The Core Mechanism: Policy Conditioning
The mathematical intuition behind preference conditioning is straightforward: we expand the policy's input space. Instead of a standard policy $\pi(a|s)$ that maps a state $s$ to an action $a$, a preference-conditioned policy operates as $\pi(a|s, w)$, where $w$ represents a vector of objective weights or preferences.
Standard RL: State (s) ───────────────────────────► [ Policy ] ──► Action (a)
▲
│ (Fixed Trade-off Baked In)
Conditioned RL: State (s) ───────┐
├─► [ Concatenate ] ──► [ Policy ] ──► Action (a)
Preference (w) ──┘ ▲
│ (Dynamically Steerable)
Early foundations for this idea were laid by Leon Barrett et al. in Learning all optimal policies with multiple criteria, where they introduced Convex Hull Value Iteration to compute the entire set of optimal linear trade-offs in a single pass. In modern deep reinforcement learning, this concept is scaled by treating the preference vector as a continuous coordinate on the Pareto front.
Haoye Lu demonstrated in Multi-Objective Reinforcement Learning: Convexity, Stationarity and Pareto Optimality that conditioning both the policy and the value function on a preference weight vector allows a single neural network to generalize across the entire trade-off space. At runtime, finding the optimal behavior simply requires computing the dot product of the value vector with the user's current weights. As a result, a single model can interpolate and execute optimal policies even for preference combinations it never explicitly saw during training, as shown by Toygun Basaklar et al. in PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm.
How the Network Learns the Frontier
Training a single network to handle conflicting goals introduces a major challenge: gradient conflict. If a preference vector points toward Objective A, the gradients pull the network weights one way; if it points toward Objective B, they pull the opposite way. Two primary strategies have emerged to make this learning process stable and efficient.
1. Cross-Preference Knowledge Sharing
Instead of treating different preferences as isolated problems, frameworks leverage the fact that experiences gained under one preference vector can inform others. In A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation, Runzhe Yang et al. introduced Envelope Q-learning, which uses a generalized Bellman equation to optimize a vector-valued reward function. This allows the agent to update its understanding of the entire Pareto frontier simultaneously from a single transition, drastically improving sample efficiency.
Alternatively, this can be framed as a meta-learning problem. Ali Ghadirzadeh et al. show in Meta-Learning for Multi-objective Reinforcement Learning that treating different preference vectors as distinct tasks allows the model to learn a high-quality "meta-policy" initialization that can instantly adapt to any specific trade-off point.
2. Structural Decoupling via Hypernetworks
To prevent internal gradient conflicts from collapsing the model's capacity, researchers have moved beyond simple state-preference concatenation. In Pareto Set Learning for Multi-Objective Reinforcement Learning, Erlong Liu et al. utilize a hypernetwork that takes the preference vector as an input and dynamically generates the weights of a separate policy network. By separating the preference-weighting function from the state-action architecture, the system optimizes the model structure for specific trade-offs without causing destructive interference in the shared layers.
Applications Across Domains
The ability to steer an agent's behavior at inference time has made preference conditioning a vital tool across a wide variety of machine learning fields.
Large Language Model Alignment
Standard Direct Preference Optimization (DPO) collapses diverse human values into a single average model, often causing an "alignment tax" where optimizing for safety degrades helpfulness. To combat this, Haoxiang Wang et al. developed Directional Preference Alignment (DPA) in Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards, which treats user preferences as unit vectors in a multi-objective reward space.
Similarly, Yiju Guo et al. introduced Controllable Preference Optimization (CPO) in Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment, utilizing explicit preference tokens (such as <Helpfulness:5>) to allow users to arithmetically tune model behavior at decoding time without retraining.
Robotics and Physical Systems
In physical environments, operational constraints often change based on context. For example, an autonomous delivery robot needs to move quickly in industrial zones but quietly in residential areas.
- In Minimizing Acoustic Noise: Enhancing Quiet Locomotion for Quadruped Robots in Indoor Applications, Zhanxiang Cao et al. integrated a scalar control parameter directly into a quadruped's policy input, enabling real-time adjustment between locomotion speed and acoustic noise impact.
- Yuyou Zhang et al. expanded on this in QuietPaw: Learning Quadrupedal Locomotion with Versatile Noise Preference Alignment by utilizing decomposed value representations, allowing a single robot to transition seamlessly into a "stealth mode" across varying terrains.
Economic and Industrial Optimization
The framework extends naturally to complex simulation and design tasks where priorities shift dynamically:
- Market Simulation: Leo Ardon demonstrated in Towards a fully RL-based Market Simulator that conditioning financial agents on reward-shaping parameters allows them to dynamically interpolate between profit maximization and market share.
- Healthcare Scheduling: In Optimizing risk-based breast cancer screening policies with reinforcement learning, Adam Yala et al. developed a preference-conditioned framework called Tempo, enabling healthcare providers to adjust patient screening frequencies dynamically based on shifting clinical resource constraints.
By shifting the optimization target from a single static point to an entire steerable manifold, preference-conditioned policies provide a scalable, elegant solution to the pluralistic nature of real-world objectives.
Key Foundational Work
Learning all optimal policies with multiple criteria (2008) — contributed 2 concept elements to this direction.