Reinforcement Learning with Vector Valued Rewards

33 concepts33 papers1 proposal

In standard reinforcement learning (RL), an agent's goal is to maximize a single scalar reward signal accumulated over time. This formulation is elegant and mathematically convenient, but it forces a fundamental compromise: it assumes that all competing human desires, operational constraints, and environmental trade-offs can be compressed into a single real number.

In practice, real-world tasks are inherently multi-objective. A autonomous vehicle must balance speed against safety; a delivery network must balance fuel costs against delivery times; an AI assistant must balance helpfulness against safety constraints. Forcing these distinct dimensions into a single scalar value creates severe limitations.

Multi-Objective Reinforcement Learning (MORL) addresses this by replacing scalar rewards with vector-valued reward signals. By treating rewards as vectors, agents can preserve the underlying trade-offs of a environment throughout the learning process, discovering an entire frontier of optimal behaviors rather than a single compromised policy.

The Scalarization Trap

The conventional way to handle multiple objectives is "premature scalarization"—multiplying each objective by a fixed weight and summing them up to create a single scalar reward:

$$R_{scalar} = w_1 R_{\text{speed}} + w_2 R_{\text{safety}}$$

As argued by Hayes et al., this approach suffers from the "scalarization trap." If the relative importance of speed versus safety changes at runtime, an agent trained on a fixed scalar reward must be completely retrained from scratch.

Furthermore, scalarization fundamentally destroys information. Vamplew et al. demonstrated that certain complex human preferences, such as lexicographic orderings (e.g., "maximize speed, but only if safety is strictly above a critical threshold"), cannot be mathematically expressed by any linear combination of scalar rewards. Collapsing these goals prematurely leads to systemic vulnerabilities like reward hacking, where the agent exploits a poorly weighted scalar function to maximize one objective at the complete expense of another (Vamplew et al.).

The historical roots of moving beyond scalar feedback trace back to early connectionist architectures. Schmidhuber proposed utilizing multi-dimensional reinforcement vectors to handle complex, non-Markovian environments, recognizing that distinct feedback units are necessary for granular credit assignment across competing goals.

Vectorizing the Bellman Equation

To preserve the independence of distinct goals, we redefine the reward signal as a vector $\mathbf{r}(s,a) \in \mathbb{R}^d$, where each dimension $i \in \{1, \dots, d\}$ tracks a separate objective. Consequently, the value function $\mathbf{Q}(s,a)$ also becomes a vector representing the expected cumulative return for each objective:

$$\mathbf{Q}(s,a) = \mathbb{E} \left[ \sum_{t=0}^{\infty} \gamma^t \mathbf{r}(s_t, a_t) \;\middle|\; s_0=s, a_0=a \right]$$

This shift introduces a major mathematical challenge: How do we compute a maximum over vectors? In standard RL, the Bellman optimality equation relies on the $\max$ operator to pick the best action. But vectors are only partially ordered; you cannot definitively say whether the vector $[10, 2]$ (high speed, low safety) is "greater than" $[2, 10]$ (low speed, high safety).

To solve this, MORL algorithms replace the scalar $\max$ operator with operations designed to track the Pareto frontier—the set of all solutions that cannot be improved in any single objective without degrading another (Chen et al., Santos et al.).

Convex Hull Value Iteration

One foundational approach to navigating this partial ordering is Convex Hull Value Iteration, introduced by Barrett et al.. Instead of maintaining a single value for each state, the algorithm maintains a set of value vectors representing the convex hull of all possible optimal outcomes.

During a Bellman backup, the algorithm computes the Minkowski sum of the current state's rewards and the downstream convex hulls. By pruning dominated vectors, the agent maintains the minimal set of policies that are optimal under any possible linear preference weighting. At runtime, when a user specifies a preference weight vector $\mathbf{w}$, the agent can instantly select the optimal policy from its precomputed hull without needing a single step of retraining.

Standard RL Backup:
[Scalar Reward] + γ * max(Scalar Values)  ---> Single Scalar Value

Vector-Valued Backup:
[Vector Reward] + γ * Hull(Vector Values) ---> Convex Hull of Optimal Vectors

Scaling to Deep Networks and Continuous Preferences

While tabular methods manage explicit sets of vectors, deep reinforcement learning requires continuous representations that can scale to high-dimensional state spaces.

Envelope Q-Learning

Yang et al. introduced Envelope Q-learning to extend vector-valued rewards to deep networks. Instead of learning separate networks for different preferences, they train a single network that takes both the state $s$ and a preference vector $\mathbf{w}$ as inputs.

The core innovation lies in the loss function, which optimizes the vectorized value function relative to the convex envelope of the next state's values. By updating the network across a distribution of sampled preferences simultaneously, the network learns the geometric relationships between different objectives. This allows the agent to achieve few-shot adaptation to entirely new preference profiles at runtime.

Building on this concept, Basaklar et al. developed PD-MORL, which incorporates a cosine similarity metric directly into the optimization process. This forces the network's vector-valued outputs to align precisely with the user's requested preference direction, maximizing the diversity and hypervolume of the discovered Pareto frontier while using a fraction of the parameters required by multi-network ensembles.

Offline Vector Conditioning

The utility of vector-valued rewards extends naturally to offline reinforcement learning, where agents must learn purely from static, pre-recorded datasets. Zhu demonstrated that conditioning offline sequence models (such as Decision Transformers) on a vector of desired returns, rather than a scalar target, provides a significantly more informative training signal. The vectorized feedback allows the model to pinpoint exactly which objective is suffering a deficit at any point in a trajectory, enabling precise, Pareto-efficient corrections during offline policy synthesis.

Beyond Expected Values: Multivariate Distributions

The frontier of vector-valued reward signals moves beyond tracking just the expected vector returns to tracking their full joint probability distributions. In many critical applications, objectives are correlated; for instance, a high-speed maneuver might drastically increase the variance (risk) of the safety objective.

Wiltzer et al. established the theoretical foundations for Multivariate Distributional Reinforcement Learning. By developing particle-based and categorical operators capable of propagating multi-dimensional probability distributions through Bellman updates, they proved that agents can reliably learn the full joint distribution of vector-valued signals. This allows decision-makers to evaluate policies not just by their average trade-offs, but by their multi-dimensional risk profiles, opening the door to highly nuanced, risk-aware deployment in complex human environments.

Key Foundational Work

Reinforcement Learning in Markovian and Non-Markovian Environments (1990) — contributed 2 concept elements to this direction.
Learning all optimal policies with multiple criteria (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_vector_valued_reward_signals_4c287e53,
  title = {vector-valued reward signals},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vector-valued-reward-signals/rdr_4c287e53984a4f1896f2261436a87f80},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_vector_valued_reward_signals_4c287e53_v1,
  title = {vector-valued reward signals},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vector-valued-reward-signals/rdr_4c287e53984a4f1896f2261436a87f80/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake