In traditional reinforcement learning (RL), we typically collapse all our desires for an agent—speed, safety, energy efficiency—into a single scalar reward. This approach forces designers to meticulously tune weights (e.g., $R = 0.7 \cdot \text{goal} + 0.3 \cdot \text{safety}$), a process that is often brittle and prone to "reward hacking," where an agent finds a way to maximize the sum while violating the spirit of the task.
A growing body of research suggests a more intuitive alternative: Lexicographic Reinforcement Learning. Instead of summing rewards, this framework ranks them in a strict hierarchy of importance. By treating objectives like a dictionary—where the first letter determines the order unless there is a tie—agents can prioritize safety or logical "promises" without sacrificing performance on secondary tasks.
The Filtering Intuition
The core mechanism of lexicographic optimization is a recursive filtering of the action space. As Joar Skalse, Lewis Hammond, and their colleagues describe in Lexicographic Multi-Objective Reinforcement Learning, the agent does not simply pick the action with the highest total value.
Instead, the agent first identifies the set of actions that are optimal (or near-optimal) for the highest-priority objective. If multiple actions satisfy this primary goal, the agent then looks at the second-priority objective to break the tie among those specific actions. This process continues down the hierarchy. This "slack" or tolerance approach, also explored by Daniel Jarne Ornia in Observational Robustness and Invariances in Reinforcement Learning via Lexicographic Objectives, allows an agent to trade a tiny fraction of its primary performance for a massive gain in a secondary metric, such as robustness to sensor noise.
Solving the Postponement Problem
One of the most difficult challenges in RL is ensuring an agent fulfills long-term "liveness" requirements—promises that something good will eventually happen. In standard MDPs, an agent might infinitely delay a necessary task to avoid a small immediate cost.
Mateo Perez, Ashutosh Trivedi, and Fabio Somenzi address this through Omega-Regular Decision Processes (ODPs). In Omega-Regular Decision Processes, they show that complex logical specifications (like "always eventually reach the charger") can be reduced to a lexicographic optimization problem. By prioritizing the probability of fulfilling these logical promises over the maximization of discounted rewards, the agent is mathematically prevented from "slacking off." It must satisfy its high-level behavioral constraints before it is allowed to worry about accumulating scalar points.
Human-in-the-Loop and Safety
Lexicographic structures also provide a natural interface for human guidance. In Augmenting Markov Decision Processes with Advising, Loïs Vanhée and colleagues demonstrate how human "nudges" can be integrated as top-level constraints. By marking certain states as forbidden or desired, the human operator sets a primary objective that the robot must respect before it attempts to optimize its own internal goal-seeking behavior.
This hierarchy is particularly useful in safety-critical environments. Rather than hoping a high penalty for collisions will keep a robot safe, designers can place "Avoid Collisions" at the top of the lexicographic stack. As Bo Xue and Yuanyu Wan show in Multi-objective Linear Reinforcement Learning with Lexicographic Rewards, this approach even allows for provable regret guarantees in linear environments, providing a rigorous foundation for balancing conflicting goals like safety versus speed.
Beyond Episodic Resets
Recent extensions of this work have moved into "continuing" environments, where an agent lives a single, uninterrupted lifetime. Milad Kazemi and Alvaro Velasquez in Average Reward Reinforcement Learning for Omega-Regular and Mean-Payoff Objectives demonstrate that lexicographic rewards can be adapted to average-reward settings. This ensures that even in infinite-horizon scenarios, the agent prioritizes formal behavioral requirements (liveness) as a primary goal, maximizing efficiency only within the subset of policies that are already "legal" or "safe."
By replacing the "weighted sum" with a "prioritized list," lexicographic RL offers a path toward agents that are more predictable, easier to constrain, and better aligned with complex human intentions.