Lexicographic Reward Prioritization

31 concepts25 papers2 proposals

When designing reward functions for reinforcement learning (RL) agents, we often find ourselves balancing competing objectives. For an autonomous vehicle, we want it to reach its destination quickly, but we also want it to obey traffic laws and avoid collisions.

The traditional way to handle multiple objectives is scalarization: combining them into a single reward function using a weighted sum:

$$R_{\text{total}} = w_1 R_{\text{safety}} + w_2 R_{\text{speed}} + w_3 R_{\text{comfort}}$$

While simple, scalarization suffers from a fundamental flaw: it allows for dangerous trade-offs. If the reward for speed is high enough, or if an agent accumulates enough small rewards over a long horizon, it can mathematically justify a catastrophic safety violation. This vulnerability leads to "specification gaming," where agents exploit the weights of a scalar reward function to maximize performance at the cost of critical constraints.

Lexicographic reward prioritization eliminates these trade-offs by replacing scalar weights with a strict, categorical hierarchy.

The Dictionary Metaphor: Lexicographic Optimality

The core intuition behind this approach is exactly how words are ordered in a dictionary. When comparing the words "Apple" and "Banana", we look only at the first letter. Because "A" comes before "B", the remaining letters are completely irrelevant. We only look at the second letter if the first letters are identical (e.g., "Apple" vs. "Apricot").

As formalized by Skalse et al., lexicographic multi-objective reinforcement learning applies this exact principle to policies. A policy $\pi$ is considered strictly better than a policy $\pi'$ if, at the highest-priority objective where their expected rewards differ, $\pi$ yields a higher value.

Scalarization (Weighted Sum):
[ Safety Reward ] + [ Speed Reward ] + [ Comfort Reward ] = Single Score (Allows Trade-offs)

Lexicographic Prioritization:
Step 1: Maximize [ Safety ]
Step 2: Within the safest policies, Maximize [ Traffic Laws ]
Step 3: Within those lawful policies, Maximize [ Speed ]

By enforcing this categorical order, a lower-priority objective (like speed) is only optimized within the subset of policies that already maximize all higher-priority objectives (like collision avoidance). This ensures that safety or formal mission constraints are treated as non-negotiable requirements rather than flexible preferences.

Algorithmic Paradigms for Enforcing Hierarchy

To implement lexicographic prioritization without collapsing the objectives into a single scalar, researchers utilize three primary algorithmic mechanisms.

1. Gating and Satisfaction Thresholds

In many real-world tasks, we do not need to maximize a safety metric to infinity; we simply need to satisfy a specific threshold (e.g., keeping the probability of a crash below a strict bound).

Li et al. utilize satisfaction thresholds to treat safety as a hard constraint, mimicking human decision-making where secondary goals are only pursued once a baseline of safety is guaranteed.

This gating mechanism is also highly effective at preventing reward hacking in non-physical environments. For instance, when training Multimodal Large Language Models (MLLMs) for spatial reasoning, Tu et al. implement a lexicographic gate: the model must satisfy basic structural constraints—such as outputting the correct text format—before it can unlock any rewards for fine-grained spatial accuracy. This prevents the model from over-optimizing for advanced metrics while failing the primary task.

2. Decoupled Value Functions and Bi-level Optimization

Standard RL algorithms collapse all environmental feedback into a single state-value function ($Q$-function). To prevent lower-priority objectives from corrupting high-priority signals, the hierarchy can be maintained by decoupling the underlying value functions.

Hammond et al. maintain entirely separate value functions for formal task satisfaction and execution efficiency. By prioritizing the satisfaction objective during policy updates, the system ensures that agents first meet formal logic requirements before optimizing for speed or resource consumption.

In continuous domains, Fan et al. operationalize this via a bi-level optimization framework called MAXSAFE. The algorithm first minimizes the probability of entering unsafe states, and subsequently maximizes performance rewards exclusively within the subset of policies identified as maximally safe.

3. Mathematical Scaling (Power-of-Two Encodings)

Can we achieve strict lexicographic ordering within standard, single-reward RL architectures? Yes, by scaling the rewards for each tier exponentially so that lower tiers can never mathematically overtake higher tiers.

Tůmová et al. demonstrate that assigning rewards to specifications using a power-of-two scaling factor guarantees a strict lexicographic hierarchy. Because the maximum possible cumulative reward from all lower-priority objectives combined is strictly less than the penalty for a single high-priority violation, the agent can never choose to game the system.

Cordeiro et al. apply a similar mathematical boundary to tree-search algorithms, ensuring that a single critical mission objective yields a higher reward than the cumulative sum of all non-critical tasks combined, preventing risky trade-offs during planning under uncertainty.

Applications Across Domains

The shift from weighted trade-offs to strict hierarchies has proven critical across several distinct domains facing safety and alignment challenges.

Autonomous Driving and Robotics

In autonomous driving, physical safety must always override traffic rules, which in turn must override passenger comfort. Wolf et al. and Veronese et al. demonstrate that structuring rewards into this explicit hierarchy prevents agents from compromising physical safety to achieve efficiency goals.

Furthermore, when systems encounter unexpected environments or find themselves in an unavoidable violation state, hierarchical prioritization dictates a clear recovery path. As shown by Ganai et al. and Zhou et al., when an agent enters a high-risk zone, a safety-first override temporarily sacrifices all performance rewards, focusing the optimization entirely on returning the agent to a feasible, safe state.

LLM Alignment and Security

The tension between helpfulness and safety in Large Language Models is another instance of multi-objective conflict. Adversarial "jailbreaking" attacks exploit this by embedding harmful requests inside a context that appeals to the model's instruction-following nature.

To defend against this, Zhang et al. introduce hierarchical goal prioritization during inference, explicitly instructing the model to favor safety constraints over helpfulness instructions. This provides a clear decision rule for ambiguous or adversarial prompts.

When scaling LLMs to autonomous agents that interact with tools and file systems, He et al. establish that enforcing a strict hierarchy where system-defined safety constraints override user-generated commands is a vital design principle to prevent external adversarial prompts from hijacking the agent's behavior.

Key Foundational Work

Adaptive Behavior Generation for Autonomous Driving using Deep Reinforcement Learning with Compact Semantic States (2018) — contributed 2 concept elements to this direction.
Minimum-violation LTL Planning with Conflicting Specifications (2013) — contributed 2 concept elements to this direction.
Severity-sensitive norm-governed multi-agent planning (2018) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_reward_prioritization_f19e0c49,
  title = {hierarchical reward prioritization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-reward-prioritization/rdr_f19e0c490f734657af99528d870995fb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_hierarchical_reward_prioritization_f19e0c49_v1,
  title = {hierarchical reward prioritization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-reward-prioritization/rdr_f19e0c490f734657af99528d870995fb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake